Redis settings in container not the same as production

[peterjaap]

We're running into OOM errors thrown by Redis; OOM command not allowed when used memory > 'maxmemory'.

During investigation, it turned out the Hypernode Docker container has Redis but it's maxmemory is pitifully low;

$ sudo cat /etc/redis/redis.conf | grep maxmemory
maxmemory 128mb

On our production Hypernodes, this is considerably higher;

$ cat /etc/redis/redis.conf | grep maxmem
maxmemory 4096mb

I tried adjusting this in /etc/redis/redis.conf on the test Hypernode, but I'm not allowed to restart Redis due to a ulimit policy;

$ sudo /etc/init.d/redis-server restart
Stopping redis-server: redis-server.
Starting redis-server: /etc/init.d/redis-server: 51: ulimit: error setting limit (Operation not permitted)
redis-server.

I also can't change the maxmemory-policy from volatile-rlu to allkeys-lru to have Redis just evict whatever is needed when it's full but I can't change that either since I can't restart Redis.

I see a few solutions;

• Allow us to restart Redis so we can set our own configuration
• Raise the maxmemory to match the production Hypernodes (probably lowest common denominator is 1GB for the Falcon S)

[vdloo]

Hi @peterjaap, there is no proper init system (no systemd, no sysvinit) in the container. So you can't do systemctl restart or service restart. The services are all started by what's in /etc/my_init.d. For example:

root@015e8fdb56b9 /etc/my_init.d # grep redis-server 60_restart_services.sh 
pkill -f /usr/bin/redis-server || /bin/true
/usr/bin/screen -S hypernode_service_redis -d -m /usr/bin/redis-server /etc/redis/redis.conf --daemonize no

So to restart Redis you can do:

root@015e8fdb56b9 /etc/my_init.d # pkill -f /usr/bin/redis-server 
root@015e8fdb56b9 /etc/my_init.d # redis-cli info
Could not connect to Redis at 127.0.0.1:6379: Connection refused
root@015e8fdb56b9 /etc/my_init.d # /usr/bin/screen -S hypernode_service_redis -d -m /usr/bin/redis-server /etc/redis/redis.conf --daemonize no
root@015e8fdb56b9 /etc/my_init.d # redis-cli info
# Server
redis_version:6.0.15

and you can attach to the redis process with for example:

# screen -x hypernode_service_redis

Anyway we'll also look into this 128 maxmemory limit, I think we can come up with a better default for this.

[peterjaap]

Right thanks, that worked so far! Raising the limit would be great however, so I don't have to put this into our deployment script.

[tdgroot]

Hi @peterjaap, we'll release a new docker image next week with the default redis maxmemory set to 4096. Letting you know when that's done!

[peterjaap]

Aaaawesome!

[tdgroot]

Hi @peterjaap,

Unfortunately we didn't get to releasing it last week and this week we had limited occupation in the systems team, but we got to releasing the new Hypernode Docker image this afternoon.

So you can pull in the new image version!