search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
16.95k stars 8.14k forks source link

chroot can not read from /dev/urandom #9549

Open schoentoon opened 1 year ago

schoentoon commented 1 year ago

This is basically #8680 but unlike what we thought at the time, it doesn't seem fixed on newer kernel/Fedora CoreOS/cri-o.

What happened:

When starting the controller it errors with the following.

2023/01/27 16:17:52 [warn] 37#37: *1 [lua] lua_ingress.lua:25: get_seed_from_urandom(): failed to open /dev/urandom: /dev/urandom: Permission denied, context: init_worker_by_lua*
2023/01/27 16:17:52 [warn] 37#37: *1 [lua] lua_ingress.lua:59: randomseed(): failed to get seed from urandom, context: init_worker_by_lua*

What you expected to happen:

I expected the controller to start without errors.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller Release: v1.5.1 Build: d003aae913cc25f375deb74f898c7f3c65c06f05 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:40:09Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

How to reproduce this issue:

Anything else we need to know:

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
Volatus commented 1 year ago

@schoentoon Could you check how the filesystems are mounted on the container? If it is mounted with the nodev option, this would inhibit block and character special device operations.

schoentoon commented 1 year ago

It looks like / is, but /dev isn't. Not sure whether that matters or not.

$ kubectl exec -it --namespace ingress-nginx ingress-nginx-controller-598fcf4865-9bnks -- /bin/sh
/chroot/etc/nginx $ mount
overlay on / type overlay (rw,nodev,relatime,seclabel,lowerdir=/var/lib/containers/storage/overlay/l/H66SR4LHSWGRWJHJ5GYUT37B5L:/var/lib/containers/storage/overlay/l/J4AA262VH4LRDGBZOULIGKXQSB:/var/lib/containers/storage/overlay/l/DYQ3P6K7OWCGFLX62WPH4UVG7Z:/var/lib/containers/storage/overlay/l/YCM7KTR6RLZLKITYVXYYDEVUUI:/var/lib/containers/storage/overlay/l/4E3CUFTIHKEBCGFO2CYSRS7ZJT:/var/lib/containers/storage/overlay/l/FVIVWB7OI525NAF6WA2QAXDXTQ:/var/lib/containers/storage/overlay/l/7NMUXOIFZQHROTGP3ZGAODCWIN:/var/lib/containers/storage/overlay/l/DCY6VIAHMZA3HVGXKMT3GOEHPL:/var/lib/containers/storage/overlay/l/NMZHSQOG3TTBWQ7LE4PPFLCIDD:/var/lib/containers/storage/overlay/l/XRQ7MBGRTVUDGV4ZLLFNN54CQF:/var/lib/containers/storage/overlay/l/SNB5TSC2VMFCG5I2RQ5P6AYOVZ:/var/lib/containers/storage/overlay/l/OP5AT6VAXU6TPHKPVB45QAWAQ2:/var/lib/containers/storage/overlay/l/34IDGNNUAIN25QDDEOFFI7BQPM,upperdir=/var/lib/containers/storage/overlay/f41713e7727fc9d9933adae5a73785a316e19ee960a4fc8c8002ce5d59c267c8/diff,workdir=/var/lib/containers/storage/overlay/f41713e7727fc9d9933adae5a73785a316e19ee960a4fc8c8002ce5d59c267c8/work,metacopy=on,volatile)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=666)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,seclabel)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k,inode64)
tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/.containerenv type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
/dev/vda4 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/vda4 on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /usr/local/certificates type tmpfs (ro,relatime,seclabel,size=3384612k,inode64)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel,size=3384612k,inode64)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,seclabel,inode64)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/keys type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,seclabel,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,seclabel,inode64)
/chroot/etc/nginx $
Volatus commented 1 year ago

It looks like / is, but /dev isn't. Not sure whether that matters or not.

$ kubectl exec -it --namespace ingress-nginx ingress-nginx-controller-598fcf4865-9bnks -- /bin/sh
/chroot/etc/nginx $ mount
overlay on / type overlay (rw,nodev,relatime,seclabel,lowerdir=/var/lib/containers/storage/overlay/l/H66SR4LHSWGRWJHJ5GYUT37B5L:/var/lib/containers/storage/overlay/l/J4AA262VH4LRDGBZOULIGKXQSB:/var/lib/containers/storage/overlay/l/DYQ3P6K7OWCGFLX62WPH4UVG7Z:/var/lib/containers/storage/overlay/l/YCM7KTR6RLZLKITYVXYYDEVUUI:/var/lib/containers/storage/overlay/l/4E3CUFTIHKEBCGFO2CYSRS7ZJT:/var/lib/containers/storage/overlay/l/FVIVWB7OI525NAF6WA2QAXDXTQ:/var/lib/containers/storage/overlay/l/7NMUXOIFZQHROTGP3ZGAODCWIN:/var/lib/containers/storage/overlay/l/DCY6VIAHMZA3HVGXKMT3GOEHPL:/var/lib/containers/storage/overlay/l/NMZHSQOG3TTBWQ7LE4PPFLCIDD:/var/lib/containers/storage/overlay/l/XRQ7MBGRTVUDGV4ZLLFNN54CQF:/var/lib/containers/storage/overlay/l/SNB5TSC2VMFCG5I2RQ5P6AYOVZ:/var/lib/containers/storage/overlay/l/OP5AT6VAXU6TPHKPVB45QAWAQ2:/var/lib/containers/storage/overlay/l/34IDGNNUAIN25QDDEOFFI7BQPM,upperdir=/var/lib/containers/storage/overlay/f41713e7727fc9d9933adae5a73785a316e19ee960a4fc8c8002ce5d59c267c8/diff,workdir=/var/lib/containers/storage/overlay/f41713e7727fc9d9933adae5a73785a316e19ee960a4fc8c8002ce5d59c267c8/work,metacopy=on,volatile)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=666)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,seclabel)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k,inode64)
tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/.containerenv type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
/dev/vda4 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/vda4 on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /usr/local/certificates type tmpfs (ro,relatime,seclabel,size=3384612k,inode64)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel,size=3384612k,inode64)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,seclabel,inode64)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/keys type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,seclabel,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,seclabel,inode64)
/chroot/etc/nginx $

/dev seems mounted properly. Can you check ls -la | grep rand? And also, can you try running cat /dev/urandom | head, it's weird that you wouldn't be able to read from it as root.

Volatus commented 1 year ago

/assign

Volatus commented 1 year ago

It looks like / is, but /dev isn't. Not sure whether that matters or not.

$ kubectl exec -it --namespace ingress-nginx ingress-nginx-controller-598fcf4865-9bnks -- /bin/sh
/chroot/etc/nginx $ mount
overlay on / type overlay (rw,nodev,relatime,seclabel,lowerdir=/var/lib/containers/storage/overlay/l/H66SR4LHSWGRWJHJ5GYUT37B5L:/var/lib/containers/storage/overlay/l/J4AA262VH4LRDGBZOULIGKXQSB:/var/lib/containers/storage/overlay/l/DYQ3P6K7OWCGFLX62WPH4UVG7Z:/var/lib/containers/storage/overlay/l/YCM7KTR6RLZLKITYVXYYDEVUUI:/var/lib/containers/storage/overlay/l/4E3CUFTIHKEBCGFO2CYSRS7ZJT:/var/lib/containers/storage/overlay/l/FVIVWB7OI525NAF6WA2QAXDXTQ:/var/lib/containers/storage/overlay/l/7NMUXOIFZQHROTGP3ZGAODCWIN:/var/lib/containers/storage/overlay/l/DCY6VIAHMZA3HVGXKMT3GOEHPL:/var/lib/containers/storage/overlay/l/NMZHSQOG3TTBWQ7LE4PPFLCIDD:/var/lib/containers/storage/overlay/l/XRQ7MBGRTVUDGV4ZLLFNN54CQF:/var/lib/containers/storage/overlay/l/SNB5TSC2VMFCG5I2RQ5P6AYOVZ:/var/lib/containers/storage/overlay/l/OP5AT6VAXU6TPHKPVB45QAWAQ2:/var/lib/containers/storage/overlay/l/34IDGNNUAIN25QDDEOFFI7BQPM,upperdir=/var/lib/containers/storage/overlay/f41713e7727fc9d9933adae5a73785a316e19ee960a4fc8c8002ce5d59c267c8/diff,workdir=/var/lib/containers/storage/overlay/f41713e7727fc9d9933adae5a73785a316e19ee960a4fc8c8002ce5d59c267c8/work,metacopy=on,volatile)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=666)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,seclabel)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k,inode64)
tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/.containerenv type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
/dev/vda4 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/vda4 on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,seclabel,size=802260k,nr_inodes=819200,mode=755,inode64)
tmpfs on /usr/local/certificates type tmpfs (ro,relatime,seclabel,size=3384612k,inode64)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel,size=3384612k,inode64)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,seclabel,inode64)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/keys type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,seclabel,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,seclabel,inode64)
/chroot/etc/nginx $

What's the version of CRI-O?

schoentoon commented 1 year ago

I'm not root inside the container though. As for your questions, here you go.

/chroot/etc/nginx $ id
uid=101(www-data) gid=82(www-data) groups=82(www-data)
/chroot/etc/nginx $ cd /dev/
/dev $ ls -la | grep rand
crw-rw-rw-    1 root     root        1,   8 Feb  1 08:18 random
crw-rw-rw-    1 root     root        1,   9 Feb  1 08:18 urandom
/dev $ cd /chroot/dev/
/chroot/dev $ ls -la | grep rand
crw-rw-rw-    1 root     root        1,   8 Nov  8 22:47 random
crw-rw-rw-    1 root     root        1,   9 Nov  8 22:47 urandom
/chroot/dev $ cat /dev/urandom | head
<snip gibberish>
/chroot/dev $ cat /chroot/dev/urandom | head
cat: can't open '/chroot/dev/urandom': Permission denied
/chroot/dev $

It does seem a bit weird to me that creation date of the devices in /chroot/dev are so far in the past though, I thought those are supposed to be created by the init container?

cri-o version is 1.24.1

longwuyuan commented 1 year ago

@schoentoon do you know where/how/why that seed is getting used in lua_ingress.lua ?

schoentoon commented 1 year ago

I have no idea about that, I just run the controller as shown in the deploy folder of this repository. Only modification I made to it were for the chroot.

github-actions[bot] commented 1 year ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.