search

docker-library / memcached

Docker Official Image packaging for memcached
http://memcached.org/
BSD 3-Clause "New" or "Revised" License
132 stars 100 forks source link

memcached:1.6.31-alpine is broken and directly stops with exit code 137 on Ubuntu 20.04 (docker.io 24.0.7-0ubuntu2~20.04.1) #99

Open heidemn-faro opened 2 months ago

heidemn-faro commented 2 months ago

Environment

EC2 instance, instance type m6i.large.

ubuntu@ip-x-x-x-x$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

ubuntu@ip-x-x-x-x:~$ docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2

ubuntu@ip-x-x-x-x:~$ uname -a
Linux ip-172-31-47-70 5.15.0-1068-aws #74~20.04.1-Ubuntu SMP Tue Aug 6 19:32:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Not working

(Adding or removing the -ti options makes no difference.)

ubuntu@ip-x-x-x-x:~$ time docker run -ti memcached:1.6.31-alpine; echo $?

real    0m0.529s
user    0m0.010s
sys     0m0.014s
137

I tested the same on my local WSL2 machine (Ubuntu 20.04 with Docker version 26.1.0, build 9714adc), and there it works fine.

Other versions - working fine

It seems that only the Alpine version of 1.6.31 is broken. I didn't check if they fully work, but at least they keep running instead of aborting with an error.

ubuntu@ip-x-x-x-x:~$ docker run -ti memcached:1.6.30-alpine
# Runs fine; killed with Ctrl-C.

ubuntu@ip-x-x-x-x:~$ docker run -ti memcached:1.6.31
# Runs fine; killed with Ctrl-C.
heidemn-faro commented 2 months ago

I tried to upgrade the docker.io package, but didn't help, same result. Docker.io is the package from the official Ubuntu package repo.

Unpacking docker.io (24.0.7-0ubuntu2~20.04.1) over (20.10.21-0ubuntu1~20.04.2) ...

heidemn-faro commented 2 months ago

Interesting: After a sudo apt upgrade + reboot, it's working on the EC2 instance.

heidemn-faro commented 2 months ago

On one of the instances, I'm getting a flaky behavior:

$ docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2

$ docker ps --all
CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS                        PORTS     NAMES
e662724a7dd5   memcached:1.6.31-alpine   "docker-entrypoint.s…"   3 seconds ago    Exited (137) 3 seconds ago              charming_moser
3063083f31df   memcached:1.6.31-alpine   "docker-entrypoint.s…"   5 seconds ago    Exited (137) 5 seconds ago              stupefied_kowalevski
35d307cc45fa   memcached:1.6.31-alpine   "docker-entrypoint.s…"   15 seconds ago   Exited (137) 14 seconds ago             great_heyrovsky
e55c06298c52   memcached:1.6.31-alpine   "docker-entrypoint.s…"   23 seconds ago   Exited (137) 23 seconds ago             hardcore_lederberg
de540fd3b68e   memcached:1.6.31-alpine   "docker-entrypoint.s…"   2 minutes ago    Exited (0) 2 minutes ago                sad_wiles
b24b1370f58f   memcached:1.6.31-alpine   "docker-entrypoint.s…"   2 minutes ago    Exited (0) 2 minutes ago                distracted_golick

I've updated the same instance to docker.io (24.0.7-0ubuntu2~20.04.1) (no other packages upgraded) + did a reboot -> Works now. No it doesn't. Still flaky.

heidemn-faro commented 2 months ago

Maybe this helps: docker inspect 26e267333903

[
    {
        "Id": "26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b",
        "Created": "2024-09-11T11:48:53.694984569Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "memcached"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 137,
            "Error": "",
            "StartedAt": "2024-09-11T11:48:53.93512905Z",
            "FinishedAt": "2024-09-11T11:48:53.936962327Z"
        },
        "Image": "sha256:2aae0d824f4bc7874b3fd26fc5e61b2fe8c3fc9b58f82f3be24a6010c05ec1c1",
        "ResolvConfPath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/hostname",
        "HostsPath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/hosts",
        "LogPath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b-json.log",
        "Name": "/quizzical_mayer",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                57,
                199
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498-init/diff:/var/lib/docker/overlay2/6b4156e1c73fc9bfe3435e1267bf456b9acc9d3808a59d0f69120505c849b2e9/diff:/var/lib/docker/overlay2/0291d95600090ab3469f937846edda895c5e992e2e30b6c5f62a008ba68f0bea/diff:/var/lib/docker/overlay2/39c2192c4a4ce4e2563fc62d50c6f5bc3d06052844f4c1732265bc6442d9e1a5/diff:/var/lib/docker/overlay2/accd7f99f8d4883549fcae51b44c5fdfe5c5c69a58590f2deace34feace95267/diff:/var/lib/docker/overlay2/f4c5a0c47c04223a936fec8d1ebabeda04c4a349b6c02fe45c7810288119e6f0/diff:/var/lib/docker/overlay2/14ff7ed0dee5f2bbb4ad1d59dd9d4571fcd85f873e2387158e1f95ca89689a23/diff",
                "MergedDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498/merged",
                "UpperDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498/diff",
                "WorkDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "26e267333903",
            "Domainname": "",
            "User": "memcache",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "11211/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "MEMCACHED_VERSION=1.6.31",
                "MEMCACHED_URL=https://memcached.org/files/memcached-1.6.31.tar.gz",
                "MEMCACHED_SHA1=85e2cb9520beba71d7fc69f5717208a57facde28"
            ],
            "Cmd": [
                "memcached"
            ],
            "Image": "memcached:1.6.31-alpine",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5ae80fd89db380d40f9a3e4ac0530634c549ae7ef8a8111470e248148dce38da",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/5ae80fd89db3",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "19d8795c83c1d65cdb2a041242daca1a3f36a9007528068ab860c23495558e62",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]
heidemn-faro commented 2 months ago

Relevant diffs:

heidemn-faro commented 2 months ago

Maybe this also helps:

$ docker run -v $PWD/log:/var/log/memcached.log -ti memcached:1.6.31-alpine -vv; echo $?; cat log
slab class   1: chunk size        96 perslab   10922
slab class   2: chunk size       120 perslab    8738
slab class   3: chunk size       152 perslab    6898
slab class   4: chunk size       192 perslab    5461
slab class   5: chunk size       240 perslab    4369
slab class   6: chunk size       304 perslab    3449
slab class   7: chunk size       384 perslab    2730
slab class   8: chunk size       480 perslab    2184
slab class   9: chunk size       600 perslab    1747
slab class  10: chunk size       752 perslab    1394
slab class  11: chunk size       944 perslab    1110
slab class  12: chunk size      1184 perslab     885
slab class  13: chunk size      1480 perslab     708
slab class  14: chunk size      1856 perslab     564
slab class  15: chunk size      2320 perslab     451
slab class  16: chunk size      2904 perslab     361
slab class  17: chunk size      3632 perslab     288
slab class  18: chunk size      4544 perslab     230
slab class  19: chunk size      5680 perslab     184
slab class  20: chunk size      7104 perslab     147
slab class  21: chunk size      8880 perslab     118
slab class  22: chunk size     11104 perslab      94
slab class  23: chunk size     13880 perslab      75
slab class  24: chunk size     17352 perslab      60
slab class  25: chunk size     21696 perslab      48
slab class  26: chunk size     27120 perslab      38
slab class  27: chunk size     33904 perslab      30
slab class  28: chunk size     42384 perslab      24
slab class  29: chunk size     52984 perslab      19
slab class  30: chunk size     66232 perslab      15
slab class  31: chunk size     82792 perslab      12
slab class  32: chunk size    103496 perslab      10
slab class  33: chunk size    129376 perslab       8
slab class  34: chunk size    161720 perslab       6
slab class  35: chunk size    202152 perslab       5
slab class  36: chunk size    252696 perslab       4
slab class  37: chunk size    315872 perslab       3
slab class  38: chunk size    394840 perslab       2
slab class  39: chunk size    524288 perslab       2
<26 server listening (auto-negotiate)
<27 server listening (auto-negotiate)
137

When starting memcached:1.6.30-alpine (the version which works correctly for me), the logs look the same - <27 server listening (auto-negotiate) is the last log entry for the startup.

LaurentGoderre commented 1 week ago

It seems like error 137 is commonly associated with running out of memory so maybe you need bigger instances or you need to configure memcache to use less memory.

https://hub.docker.com/_/memcached

plotas commented 4 days ago

Environment

EC2 instance, instance type m6i.large.

ubuntu@ip-x-x-x-x$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

ubuntu@ip-x-x-x-x:~$ docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2

ubuntu@ip-x-x-x-x:~$ uname -a
Linux ip-172-31-47-70 5.15.0-1068-aws #74~20.04.1-Ubuntu SMP Tue Aug 6 19:32:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Not working

(Adding or removing the -ti options makes no difference.)

ubuntu@ip-x-x-x-x:~$ time docker run -ti memcached:1.6.31-alpine; echo $?

real    0m0.529s
user    0m0.010s
sys     0m0.014s
137

I tested the same on my local WSL2 machine (Ubuntu 20.04 with Docker version 26.1.0, build 9714adc), and there it works fine.

Other versions - working fine

It seems that only the Alpine version of 1.6.31 is broken. I didn't check if they fully work, but at least they keep running instead of aborting with an error.

ubuntu@ip-x-x-x-x:~$ docker run -ti memcached:1.6.30-alpine
# Runs fine; killed with Ctrl-C.

ubuntu@ip-x-x-x-x:~$ docker run -ti memcached:1.6.31
# Runs fine; killed with Ctrl-C.
heidemn-faro commented 2 days ago

It seems like error 137 is commonly associated with running out of memory so maybe you need bigger instances or you need to configure memcache to use less memory.

https://hub.docker.com/_/memcached

Thanks for your reply @LaurentGoderre, but not sure if you fully read my bug report ;-)

I was using m6i.large with 8 GB of RAM. I think 8 GB is a common configuration for cloud servers, and the image should work on such common configurations with the default config.

And as I said, both memcached:1.6.30-alpine and memcached:1.6.31 work fine. Only memcached:1.6.31-alpine fails to start, which makes me believe that something is wrong with the image.

I see there is now a 1.6.32-alpine, will give it a try soon.

tianon commented 1 day ago

The differences between the 1.6.30 image and the 1.6.31 image are extremely minimal; they should amount to minor updates to the base image/packages and the memcached version bump. If you're having an issue on 1.6.31 that does not exhibit on 1.6.30, the most likely cause is something upstream or something environmental, not something in the image (so not something we are likely to change the image to "fix"), and I would suggest the place to start your debugging would be looking at the changes that went into 1.6.31 to determine whether any of them sound like candidates for causing the behavior you're seeing.

Regardless, digging further is going to be most fruitful in a dedicated support forum, especially one with more knowledge/experience running memcached itself (irrespective of Docker).