Redis performance is poor in Docker network compared to unix sockets

[nickchomey]

I just ran a simple benchmark (redis-benchmark -q -n 1000000 -s /var/run/redis/redis.sock -t set,get,mset -P 16 -c 100) with 3 networking configurations

1. redis on host system over tcp
2. redis on host system with unix socket
3. redis in ddev container (presumably over tcp)

You can see that unix sockets are considerably faster than tcp. And there seems to be a signficant tax with the Docker Container.

I don't know much about docker networking, so I dont know what to look at or change.

Here's some links that I've found that might be helpful

https://medium.com/@jonbaldie/how-to-connect-to-redis-with-unix-sockets-in-docker-9e94e01b7acd https://www.reddit.com/r/selfhosted/comments/vf6jeg/i_used_unix_sockets_to_improve_the_performance_of/ https://help.nextcloud.com/t/solved-cant-get-redis-to-work-in-socket-mode/122830/13 https://serverfault.com/questions/1156554/bind-a-host-unix-socket-to-a-container-port

[nickchomey]

If I dont use any pipelining and just 1 client, heres the results. Interesting that docker is only slightly slower than host tcp here. Both much slower than unix sockets

[nickchomey]

Here's a real-world test - using it with a Wordpress site.

This is a very bare bones site compared to what I normally run, and the average time is usually something like 5-10ms. So this is more than 10x slower. So there's surely something else going on than just Unix sockets and docker networking. (though, I see now that it's all more than 10x slower than the initial benchmark shared above, so could be an explanation).

I have to go right now, but I'll post the same metrics if/when I can get WP to connect to Redis on the host machine, over tcp and unix. I don't see any reason why those WP metrics couldn't/shouldn't be the same sub-10ms

[nickchomey]

We can also try to get the Relay redis client/protocol working with DDEV. I use it on normal non-dockerized Ubuntu servers.

It's inherently faster than phpredis, and also has its own caching layer (which is limited with the free version, and the full version isn't at all worthwhile if you can get the average times down to 10ms - it would only make sense if they were 100ms+)

https://relay.so/

[nickchomey]

Heres the metrics with redis on host machine with tcp. Not a whole lot of difference, which was to be expected given the single-client benchmark results above.

And I just got it connected to redis on host machine via unix socket. It seems to be somewhat faster after some similar browsing to prime it - probably within the 20-40% difference range that was seen in the redis-benchmarks above.

So, implementing UNIX sockets in the docker container is perhaps worthwhile.

But its still a fair bit slower than expected (on a relative basis - obviously 0.05 seconds is immaterial...). Though, my main basis for comparison is on much faster latest-gen dedicated servers hardware at Hetzner. This could very well be just the expected performance for my laptop. I suppose I'd have to run the site on the host machine to see if docker is adding any extra overhead. I will update this issue if I end up doing that.

[nickchomey]

Here is it with Unix Sockets to the Host Machine Redis Server and Relay

Perhaps Relay is to account for what I remember! To check, here it is with the memory/cache feature turned off (the lump in the middle is when I reloaded like 10 pages all at once):

So, it does what it says!

The cache helps - though, in this case, its maybe 0.03 seconds improvement, so whether it is worth $150/month is up to you. And even without the cache (which I simulated by turning it off completely, because it drops to a low RAM amount after an hour) it is faster than phpredis.

Anyway, here's the Relay Dockerfile. The sed commands for the variables could probably be set via DDEV config.yaml environment variables.

ENV extension=relay
SHELL ["/bin/bash", "-c"]

RUN curl -fsSL "https://repos.r2.relay.so/key.gpg" | gpg --dearmor -o "/usr/share/keyrings/cachewerk.gpg" && \
    echo "deb [signed-by=/usr/share/keyrings/cachewerk.gpg] https://repos.r2.relay.so/deb $(lsb_release -sc) main" \
    | sudo tee /etc/apt/sources.list.d/cachewerk.list > /dev/null && \
    (apt-get update || true) && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-confnew" --no-install-recommends --no-install-suggests build-essential php-pear php${DDEV_PHP_VERSION}-dev && \
    apt-get install -y php${DDEV_PHP_VERSION}-relay && \
    chmod 666 /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini 
RUN sed -i 's/^;\? \?relay.maxmemory =.*/relay.maxmemory = 512M/' /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini && \
    sed -i 's/^;\? \?relay.eviction_policy =.*/relay.eviction_policy = lru/' /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini && \
    sed -i 's/^;\? \?relay.environment =.*/relay.environment = production/' /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini && \
    phpenmod ${extension} 

This Wordpress Redis plugin can be referenced for how to make use of Relay in your application. And here's the relay-specific code

I can share what I did for Unix sockets if there is interest. Though I think someone else who is more familiar with docker might need to implement it in this addon.

[rfay]

I'm not optimistic that unix sockets can be used either in Docker or in a typical production setup (as the server would usually be on a different server, just as it is in Docker). But maybe a maintainer/user of this add-on will want to follow up.

(I'm not surprised that unix sockets is faster)

[nickchomey]

I'm not optimistic that unix sockets can be used either in Docker or in a typical production setup (as the server would usually be on a different server, just as it is in Docker). But maybe a maintainer/user of this add-on will want to follow up.

(I'm not surprised that unix sockets is faster)

This is all irrelevant if someone is running Redis on a separate server - network latency would be 100-1000x more than the latency for a query to a local Redis server.

But, as shown, it can be helpful when using DDEV locally (which is essentially officially advised not to use in production). But if you are running Redis on the same production server as your application (which I absolutely intend to do, along with mariadb for the same latency reasons) - docker or not - then you could benefit from of all this as well.

I also don't suppose that Redis developed and documented support for unix sockets just for use in local development...

I'd even go so far as to say that if someone with a wordpress site is running Redis and/or mariadb on a separate server, then they don't have a clue what they're doing, because WP's architecture often results in many hundreds of queries per request. Best to scale vertically - which you can get up to nearly 200 physical cores in a server nowadays. And/or replicate the db and redis to multiple servers so that they can be colocated with the application.

Also, the links I shared in the OP seem to show that it is possible to do Unix Sockets in Docker.

And Ive partially shown it - I created a socket in the host machine and then added it to the ddev web container via a bind volume.

/<project>/.ddev/docker-compose.override.yaml

services:
    web:
        volumes:
            - /var/run/redis:/var/run/redis

[nickchomey]

I got both redis and mariadb unix sockets working with wordpress. Its dead-simple with the first link in OP. Its all just about making a socket path available to all containers via a common volume (just as we do for other paths, like the web app directory)

services:
    tmp:
        image: busybox
        command: chmod -R 777 /tmp/docker
        volumes:
            - /tmp/docker/
    web:
        volumes_from:
            - tmp
    db:
        volumes_from:
            - tmp
    redis:
        volumes_from:
            - tmp

It seems it doesn't even need to be a bind volume to the host system such as I used in the previous comment... Might be worth testing that - although, like this every project would have its own isolated socket within its own docker network, rather than all using the same host-system /tmp/docker/*.sock, which would surely create conflicts... Since it is internal to the containers' network, I wonder if it would work across servers (even though it would be slow)...?

<project-path>/.ddev/mysql/unix_socket.cnf

[mysqld]
socket                         = /tmp/docker/mysql.sock

<project_path>/.ddev/redis/network.conf

unixsocket /tmp/docker/redis.sock
unixsocketperm 777

I havent done any further benchmarking yet, but I dont really need to - its well known that sockets are 20-50% faster than TCP, and I showed that Redis Relay provides a considerable performance increase over even Unix sockets.

All the necessary snippets are shared in this thread, and they really wouldnt affect anything to add - even if people dont use them (which seems to be the general DDEV ethos for all of its images/containers). I'm happy to discuss any changes if you want a PR here or elsewhere, or feel free to incorporate them yourselves as you see fit.

[rfay]

Congratulations on the excellent experimentation and analysis! Maybe maintainer @seebeen will pick it up and run with it. I barely know what redis is.

[seebeen]

Hello Nick,

Thanks for the immense effort in benchmarking and wrangling with configs. You are 100% correct:

• sockets are the fastest for both the redis and MySQL
• Docker bridged networking adds a considerate overhead which reduces performance.

However - there are additional factors which slow redis server, but it mostly boils down to sysctl config and overrides.

I have a local branch with a similar setup as yours - socket communication with tmp dirs and bind mounts. But it has compatibility problems with colima and docker desktop on MacOS (M1) - so I disregarded it.

All in all - I believe the compabitility issues outweigh the performance benefits. Especially since DDEV is primarily a development environment.

If you have the time - I'd love to see your fork with socket implementation which passes the tests on MacOS, WSL, Linux - with both Docker and Colima. If all works, We can work together to create a config script which will give users the implementation choice - default network stack or socket connection.

Again - thank you for spending your time in order to make this extension better - and I hope you aren't dissuaded on working on it further.

P.S. If you'd like to chat / talk about finer points I'm available on DDEV discord. You can't miss me 😅

[nickchomey]

I barely know what redis is.

Its the industry-standard (though they somewhat committed suicide recently with changes to their open source licensing, so there's a lot of forks and new api-compatible competitors - perhaps I'll open a separate issue to add support for using those instead of Redis, since the same clients can connect to them seamlessly) in-memory KV store, generally used for caching of various sorts. If not used on same server, it seems somewhat pointless to use it at all as the latency gains from in-memory store would be vastly outweighed by network latency. Might as well use disk storage at that point to save money - Rails' new Solid Cache does so with Mysql/postgres/sqlite/etc... to great effect.

So, if you have Redis (and Mysql/mariadb for that matter, as I easily incorporated in my previous comment!) in use on the same server as the application, might as well use unix sockets to improve query speed - particularly with a rats nest like Wordpress that makes such an unconscionable amount of db/redis requests per php page request. And Relay is a cherry on top, regardless of TCP/Unix socket choice - its in-process cache prevents many requests from even leaving PHP, let alone the web container.

[nickchomey]

@seebeen Yeah, I had seen some documentation and discussions about how docker bridge networks are slower than using the host network, and assume that's what mostly accounts for the unexpectedly slow redis response times that started this issue. But I figured that host network simply wouldnt work at all in the context of DDEV.

Though, using Relay essentially solves this problem because not only does it communicate with Redis more quickly from PHP, but it also has its own RAM cache that is in-process with PHP itself, so the request doesn't even leave PHP at all, let alone the webserver container. In that respect, it sort of makes unix sockets irrelevant as well - though there's no reason not to use them anyway, and they're helpful for when a request misses Relay's cache. As mentioned, the RAM limit drops from unlimited to 16MB after an hour, which significantly lessens its utility as it will be constantly purging items from its cache on a LRU basis. Yet, surely this isn't a problem in a dev environment, where there's minimal requests, one (or few) users, and therefore not a whole lot of variety in cahed data. (there's also a... trick... that can be used to completely sidestep this 1-hour limit, but I'm hesitant to share it publicly out of respect for the dev's generosity in sharing Relay as a free version. It can remain a hacker's secret)

Anyway, that's unfortunate that there's compatibility issues with unix sockets on colima and mac docker desktop! But, as you suggested, perhaps there's a way we can make it work such that its an easy opt-in. I'm happy to collaborate on that.

Probably best to keep discussion here rather than Discord, so as to make it easier for others to discover, as well as force us (me) to be more concise - I tend to ramble even more on Discord, so am generally avoiding it these days.

When you say "fork", you mean of this add-on? I didn't actually fork anything - just installed it and then, in the end, manually added 4 files/snippets. I'll re-list slightly modified versions here, so as to consolidate all my prevoius ramblings into one place.

<project_path>/.ddev/docker-compose.unixsockets.yaml adds the required socket volume. Could perhaps use /tmp/unixsockets/:/tmp/unixsockets/ so that it is a bind mount instead of bridge, allowing for unix socket access from host OS as well...?

services:
    unixsockets:
        image: busybox
        command: chmod -R 777 /tmp/unixsockets
        volumes:
            - /tmp/unixsockets/ 
    web:
        volumes_from:
            - unixsockets
    db:
        volumes_from:
            - unixsockets
    redis:
        volumes_from:
            - unixsockets

<project_path>/.ddev/redis/network.conf Enables unix socket support in Redis.

unixsocket /tmp/unixsockets/redis.sock
unixsocketperm 777

ddev-projects/stfgz/.ddev/web-build/Dockerfile.relay - Installs and activates the Relay php extension in the web container, which apps would then need to connect to. Here's some code from the Wordpress redis plugin (that is developed by the same person who develops Relay) that could be used as a starting point for non-WP applications. Instructions from that plugin can be followed to configure WP to connect to Redis with Relay and Unix Sockets.

ENV extension=relay
SHELL ["/bin/bash", "-c"]

RUN curl -fsSL "https://repos.r2.relay.so/key.gpg" | gpg --dearmor -o "/usr/share/keyrings/cachewerk.gpg" && \
    echo "deb [signed-by=/usr/share/keyrings/cachewerk.gpg] https://repos.r2.relay.so/deb $(lsb_release -sc) main" \
    | sudo tee /etc/apt/sources.list.d/cachewerk.list > /dev/null && \
    (apt-get update || true) && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-confnew" --no-install-recommends --no-install-suggests build-essential php-pear php${DDEV_PHP_VERSION}-dev && \
    apt-get install -y php${DDEV_PHP_VERSION}-relay && \
    chmod 666 /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini 
RUN sed -i 's/^;\? \?relay.maxmemory =.*/relay.maxmemory = 512M/' /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini && \
    sed -i 's/^;\? \?relay.eviction_policy =.*/relay.eviction_policy = lru/' /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini && \
    sed -i 's/^;\? \?relay.environment =.*/relay.environment = production/' /etc/php/${DDEV_PHP_VERSION}/mods-available/${extension}.ini && \
    phpenmod ${extension} 

<project-path>/.ddev/mysql/unix_socket.cnf Changes the mariadb path for unix sockets (it is configured by default as /var/tmp/mysql.sock within the db container). For wordpress, just need to set define('DB_HOST', 'localhost:/tmp/unixsockets/mysql.sock'); in wp-config.php and it will connect .

[mysqld]
socket                         = /tmp/unixsockets/mysql.sock

I suppose that the mysql stuff would be better placed elsewhere, but I dont think it merits an entire add-on just for it... Likewise the unixsockets volume probably could/should be created elsewhere so that it is more general and available to other containers and volumes.

Let me know how you'd like to proceed! Any suggestions on how to include/activate these things - be it in this addon, a separate addon, or perhaps even ddev core (though I think we'd all prefer to avoid that!) - would be appreciated.

I dont see much issue with installing and activating it all by default, unless it literally breaks the ddev start image build on those OSs. If that's the case, I'm open to any suggestions on how to make it build on a conditional basis (I dont think a ddev redis relay or ddev redis unix command make a whole lot of sense - unix sockets can be used alongside tcp connections, and likewise the relay extension can be used alongside the phpredis extension. So, they should just be there in the background, ready to be optionally connected to).

If you want me to put those 4 files in a fork of this addon, i can do that.

Let me know!

[rfay]

Another thing you might want to consider if you're having trouble with redis performance is to install it in the webserver instead of as a separate service/container. Just use a Dockerfile.redis to install it there with https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/install-redis-on-linux/

And despite the suggestion there, just webimage_extra_packages: [redis] will install redis v7.

You could start it with web_extra_daemons

[nickchomey]

Thanks, nice idea. I just tried for a bit but can't (be bothered to) figure it out. Its having trouble with the config files etc... Much easier with docker and this addon. With the unix sockets and Relay, its performing extremely well.

[nickchomey]

Turns out that DDEV needs the mysql.sock file to be at /var/tmp/mysql.sock, for example to be able to do ddev export-db. So, it doesn't work to put it in the /var/tmp/unixsockets shared volume.

Its all for the best though, because here's a simpler way to get all of this working. No <project-path>/.ddev/mysql/unix_socket.cnf needed, as we can just share the /var/tmp/mysql.sock from the db container with the web container, and likewise /var/tmp/redis.sock.

services:
    web:
        volumes:
            - shared_tmp:/var/tmp
        depends_on:
            - db

    db:
        volumes:
            - shared_tmp:/var/tmp

    redis:
        volumes:
            - shared_tmp:/var/tmp
        depends_on:
            - db

volumes:
    shared_tmp:
        name: ddev-${DDEV_SITENAME}-shared_tmp
        labels:
            com.ddev.site-name: ${DDEV_SITENAME}

<project_path>/.ddev/redis/network.conf

unixsocket /var/tmp/redis.sock
unixsocketperm 777

[nickchomey]

I see that there's an active pr for making ddev get more extensible/flexible. Best to wait for that to get merged before proceeding with anything here

https://github.com/ddev/ddev/pull/6406

[nickchomey]

@unfixa1 yes, using Unix sockets is the point of this issue and was detailed in my various comments above...