search

cruizba / ubuntu-dind

A docker image based in ubuntu to run docker containers inside docker containers
Apache License 2.0
169 stars 76 forks source link

Latest release fails to seed PRNG #28

Closed kenovergard closed 3 months ago

kenovergard commented 3 months ago

Recently our pipelines began failing any jobs that used this ubuntu-dind image with the following error:

    curl: (35) Insufficient randomness

I traced this down to the update to this image...this one-liner fails with the above error:

docker run -it cruizba/ubuntu-dind:latest sh -c "apt install -y curl ; curl -fsSL https://www.ubuntu.com/"

While this works fine:

docker run -it cruizba/ubuntu-dind:jammy-26.1.3 sh -c "apt install -y curl ; curl -fsSL https://www.ubuntu.com/"

So something has clearly gone wrong with the base image for the latest release, and I imagine lots of breakages will be reported soon.

cruizba commented 3 months ago

@kenovergard this command it's working on my machine:

docker run -it cruizba/ubuntu-dind:latest sh -c "apt install -y curl ; curl -fsSL https://www.ubuntu.com/"

Could it be something related with the kernel installed on your host or maybe your docker version?

cruizba commented 3 months ago

I've tested the command on multiple host, and I am unable to reproduce it.

kenovergard commented 3 months ago

@cruizba yeah that's just a stripped-down simplified version of the "real" command we use (which is curling the GPG key for node to add a new package source to install node). Even a simple ssh-keygen fails with a similar error (PRNG is not seeded).

docker --version on the host reports Docker version 20.10.25, build b82b9f3

The host system is a hardened customization of Amazon Linux v2, uname -a as follows:

Linux ip-<redacted>.ec2.internal 4.14.343-261.564.amzn2.x86_64 #1 SMP Tue May 7 02:23:34 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

So that's probably a contributing factor, but the fact that a jammy-based image runs fine and the noble does not indicates to me an incompatibility with the latest update. I'm having my teams pin the version to jammy until the incompatibility is resolved, but this obviously means a lot of manual work for each update so risks things getting out of date without constant vigilance.

cruizba commented 3 months ago

Can you test if the original ubuntu image also fails? I am pretty sure there is some incompatibility between your host and the latest ubuntu image:

docker run -it ubuntu:24.04 sh -c "apt update && apt install -y curl ; curl -fsSL https://www.ubuntu.com/"

See: https://github.com/cruizba/ubuntu-dind/blob/master/ubuntu-noble.Dockerfile#L1

kenovergard commented 3 months ago

Yup, that tracks; ssh-keygen gets the same error; I can raise this with our SecEng team, that seems...less than ideal. Thanks for helping!

cruizba commented 3 months ago

Yes, shit happens :(.

Ping me if you solved your problem :)

kenovergard commented 3 months ago

Well I can workaround the issue for now by just using the jammy jellyfish variant of the package. The root of the problem is the host system using FIPS mode, and the latest noble numbat-based ubuntu base image not playing nice with a FIPS mode host. I suppose it would be possible to fork this repo to build a fips-compliant image, but nothing I'm going to do about it late on a Friday ;)

cruizba commented 2 months ago

@kenovergard I've created from version 27.0.1 new latest tags by ubuntu versions:

Maybe this helps you to pin a specific ubuntu version

Better late than never 😅