search

redis / docker-library-redis

Docker Official Image packaging for Redis
http://redis.io
BSD 3-Clause "New" or "Revised" License
1.11k stars 560 forks source link

Debian bookworm upgrade renders container unable to start #365

Closed boserup closed 6 months ago

boserup commented 1 year ago

The changes introduced in 7e4a3dd9d2644458a40700a96620f6f028887a25 has broken our CI pipeline (Travis CI). We began seeing errors as the recent push to the 7.0 tag happened on DockerHub https://hub.docker.com/layers/library/redis/7.0/images/sha256-178215249742b63308db1a5373a7c627714c582362f3dcd24b2eec849dc16e67?context=explore

It is worth mentioning that we resolved the issue temporarily by switching to the 7.0-bullseye image, but I still don't see a reason why the bookworm-based images should be crashing.

Starting a container using the redis:7.0 image produces the following output:

redis_1  | 1:C 15 Jun 2023 09:43:37.054 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1  | 1:C 15 Jun 2023 09:43:37.054 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1  | 1:C 15 Jun 2023 09:43:37.054 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1  | 1:M 15 Jun 2023 09:43:37.055 * monotonic clock: POSIX clock_gettime
redis_1  | 1:M 15 Jun 2023 09:43:37.055 * Running mode=standalone, port=6379.
redis_1  | 1:M 15 Jun 2023 09:43:37.055 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1  | 1:M 15 Jun 2023 09:43:37.055 # Server initialized
redis_1  | 1:M 15 Jun 2023 09:43:37.055 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1  | 1:M 15 Jun 2023 09:43:37.055 # Fatal: Can't initialize Background Jobs.
api_redis_1 exited with code 1

In docker-compose.yml the container is defined as such:

version: '3.4'
services:
  redis:
    image: redis:7.0
    hostname: redis
    ports:
      - "6379:6379"
olliemath commented 1 year ago

Seen in our piplines running 6.x as well

michaelarnauts commented 1 year ago

We also notice this. The host is running on Ubuntu 18.04 with a 4.15.0-212-generic kernel. I assume it has something to do with the kernel version or the base OS?

dstotz commented 1 year ago

Same issue in our Buildkite pipelines running version 7.0.11

jrosenwald commented 1 year ago

Seen in our pipelines with 7.0.11

yosifkit commented 1 year ago

If you are having troubles, try using the redis:*-bullseye tags. If that fixes it, then you probably need to update Docker and libseccomp on the host (and maybe possibly kernel version, but try the others first). Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it.

michaelarnauts commented 1 year ago

If you are having troubles, try using the redis:*-bullseye tags. If that fixes it, then you probably need to update Docker and libseccomp on the host (and maybe possibly kernel version, but try the others first). Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it.

I've tried updating to the latest docker version to no avail. I think it must be related to a specific kernel version.

The bullseye images are working fine.

yosifkit commented 1 year ago

I've tried updating to the latest docker version to no avail. I think it must be related to a specific kernel version.

Did you try updating libseccomp; not certain how exactly new it needs to be. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

This is similar to the update to Ubuntu focal: https://github.com/docker-library/mongo/issues/606#issuecomment-1424965831

Edit: This seems eerily similar to https://github.com/docker-library/python/issues/835

curtiscook commented 1 year ago

Hi, it would be helpful to know more details here -- is the best short-term solution to pin to use bullseye? is bullseye going to break in the next release, so should we also pin a version?

Using recent AWS AMI's, docker-compose is no longer able to bring up redis:latest which (per docs) is the defacto image.. to which I would argue is a fairly jarring change

version: "3.2"

services:
  redis:
    image: redis
    ports:
      - 6379
tianon commented 1 year ago

The best short term solution for you is going to be pinning to bullseye images explicitly, yes. However, they are no longer actively maintained/supported in this case, meaning that when there are updates to Debian or Redis you will not be receiving them, so you do want to figure out the issues with your environment and get yourself upgraded to the bookworm-based images.

mathieujobin commented 1 year ago

I have the same problem, it was working up to Wed 14th Jun 2023 at 20:43 UTC. I am simply pulling latest,

[2023-06-14T20:58:50Z] Pulling redis (redis:)...
latest: Pulling from library/redis
759700526b78: Pull complete
eb5c5a6a60f0: Pull complete
12e7d88aa422: Pull complete
8f4ed3de3123: Pull complete
0705c505f99b: Pull complete
d943febfd435: Pull complete
Digest: sha256:6a5130174e14177c3777ea34f4989a91bb5a7d7562dd48578cbfa3ad8cd53743
Status: Downloaded newer image for redis:latest

is it possible to know which tag this corresponds to ?

Update: bullseye has the same sha256

Digest: sha256:6a5130174e14177c3777ea34f4989a91bb5a7d7562dd48578cbfa3ad8cd53743
Status: Downloaded newer image for redis:bullseye
curtiscook commented 1 year ago

The best short term solution for you is going to be pinning to bullseye images explicitly, yes. However, they are no longer actively maintained/supported in this case, meaning that when there are updates to Debian or Redis you will not be receiving them, so you do want to figure out the issues with your environment and get yourself upgraded to the bookworm-based images.

My issue is that there is no clarity on what are the requirements to make the new image work. I'm happy to make updates as necessary but nothing here tells me the root cause

tr33m4n commented 1 year ago

We're in the same boat, all our pipelines started to fail yesterday when the latest version was released. We pinned to the bullseye version

image: redis:7.0.11-bullseye

Ref for anyone else coming to this issue fresh.

If this is something we need to change in our CI pipelines to support bookworm, can this be clarified?

benmod commented 1 year ago

We had the same issue on our side with the redis:6.2 image. However the strange thing is that it happened only on some of our test platform with slightly older versions of Docker. In that case updating Docker fixed the issue. I'm still puzzled by what is the root cause.

curtiscook commented 1 year ago

We had the same issue on our side with the redis:6.2 image. However the strange thing is that it happened only on some of our test platform with slightly older versions of Docker. In that case updating Docker fixed the issue. I'm still puzzled by what is the root cause.

it seems like there's some call in the build that requires the latest version of debian/docker, which wouldn't be a problem locally but the reality is the redis docker image is heavily used for CI pipelines and not production systems, where it makes sense to just stay on latest instead of pinning versions. This also makes it more jarring, since all of your builds start failing. Would be great to know a root cause so people can go back to using latest

tianon commented 1 year ago

If you want something "unversioned" that still works on systems which are not updated appropriately to run containers on newer distributions utilizing newer kernel system calls, you can use redis:bullseye, but with the caveat that it won't be updated (it's the same as using redis@sha256:6a5130174e14177c3777ea34f4989a91bb5a7d7562dd48578cbfa3ad8cd53743, which might even be a better idea so you're clearly choosing to rely on an unchanging/non-maintained image, but YMMV).

yosifkit commented 1 year ago

Root cause: it is Docker with libseccomp so a newer syscall used in Debian Bookworm packages/libs is being blocked.

libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).

mgrahamx commented 11 months ago

It's not possible to update libseccomp on Mac it seems, so I just moved to bullseye images and that worked.

tianon commented 11 months ago

Mac OS cannot natively run Linux containers, so that would mean you need to update whichever system/VM is managing your Linux containers (Docker Desktop, etc). :heart: