search

aws-samples / amazon-eks-ami-rhel

This is a Red Hat Enterprise Linux specific forked version of the official awslabs amazon-eks-ami repository.
https://aws-samples.github.io/amazon-eks-ami-rhel/
MIT No Attribution
14 stars 12 forks source link

Error while installing nodeadm #12

Open rishabhjain62 opened 4 days ago

rishabhjain62 commented 4 days ago

We are trying to create image - but getting this error

2024-11-12T18:14:15+05:30: amazon-ebs: WARN[0036] failed to clean up container networking: "f664476f40cc44536c37d35e89f934e1e49b2624ee2564e4e5d9ce52b3481a35" error="hosts-store error\nnot found\nstat /var/lib/nerdctl/1935db59/etchosts/k8s.io/f664476f40cc44536c37d35e89f934e1e49b2624ee2564e4e5d9ce52b3481a35/meta.json: no such file or directory"

2024-11-12T18:14:15+05:30: amazon-ebs: FATA[0036] failed to start binary process with cmdArgs [/usr/bin/nerdctl _NERDCTL_INTERNAL_LOGGING /var/lib/nerdctl/1935db59]: fork/exec /usr/bin/nerdctl: operation not permitted

bradwatsonaws commented 4 days ago

Hi @rishabhjain62 - this looks to be a local permissions related issue. Do you have SELinux enabled on your RHEL AMI? If so, can you check /var/log/messages and /var/log/audit/audit.log files for SELinux denials? There are commands in the install-worker.sh script to account for SELinux but they could vary with each environment. Instructions on checking SELinux denials here: https://www.redhat.com/en/blog/selinux-denial2

Have you modified the install-worker.sh or install-nodeadm.sh scripts at all or you using them as is?

This looks similar to this issue with nerdctl - https://github.com/containerd/nerdctl/issues/2940

rishabhjain62 commented 4 days ago

Hi @bradwatsonaws - I have updated install-worker.sh Commented out - 

# Update the OS to begin with to catch up to the latest packages.
#sudo dnf update -y

and added GPG key for installing amazon-ssm, since it was giving error. Base image used for this is a custom image with hardening(fapolicyd, selinux, disa stig)

No denials in audit.log or messages

node=localhost type=SYSCALL msg=audit(1731432064.842:3429): arch=c000003e syscall=263 success=no exit=-21 a0=8 a1=c001283c19 a2=0 a3=0 items=2 ppid=4203 pid=4205 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=16 comm="nerdctl" exe="/usr/bin/nerdctl" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="delete"ARCH=x86_64 SYSCALL=unlinkat AUID="ec2-user" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" node=localhost type=SYSCALL msg=audit(1731432064.842:3430): arch=c000003e syscall=263 success=yes exit=0 a0=8 a1=c001283c27 a2=200 a3=0 items=2 ppid=4203 pid=4205 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=16 comm="nerdctl" exe="/usr/bin/nerdctl" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="delete"ARCH=x86_64 SYSCALL=unlinkat AUID="ec2-user" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" node=localhost type=SYSCALL msg=audit(1731432064.843:3431): arch=c000003e syscall=263 success=yes exit=0 a0=3 a1=c00129e280 a2=200 a3=0 items=2 ppid=4203 pid=4205 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=16 comm="nerdctl" exe="/usr/bin/nerdctl" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="delete"ARCH=x86_64 SYSCALL=unlinkat AUID="ec2-user" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root

Nov 12 17:20:51 ip-10-1-48-206 zlogger[4198]: [/dev/pts/1:ec2-user:1000]: [47 12/11/24 17:20:51]: [sudo nerdctl run --rm --network host --workdir /workdir --volume ./nodeadm-master:/workdir public.ecr.aws/eks-distro-build-tooling/golang:1.23 make build] Nov 12 17:20:53 ip-10-1-48-206 systemd[1]: var-lib-containerd-tmpmounts-containerd\x2dmount1891135340.mount: Succeeded. Nov 12 17:21:04 ip-10-1-48-206 containerd[1143]: time="2024-11-12T17:21:04.679206557Z" level=info msg="ImageCreate event name:\"public.ecr.aws/eks-distro-build-tooling/golang:1.23\"" Nov 12 17:21:04 ip-10-1-48-206 containerd[1143]: time="2024-11-12T17:21:04.721474953Z" level=info msg="ImageCreate event name:\"sha256:b644f8e72e9c6748ebdb3b42dfb2f12a3d5fb954526d75ba62ad5105f59d7011\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}" Nov 12 17:21:04 ip-10-1-48-206 containerd[1143]: time="2024-11-12T17:21:04.723007361Z" level=info msg="ImageUpdate event name:\"public.ecr.aws/eks-distro-build-tooling/golang:1.23\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}" Nov 12 17:21:04 ip-10-1-48-206 systemd[1]: tmp-initialC1694012664.mount: Succeeded. Nov 12 17:21:04 ip-10-1-48-206 systemd[1]: tmp-containerd\x2dmount1830586336.mount: Succeeded. Nov 12 17:21:05 ip-10-1-48-206 systemd[1]: tmp-containerd\x2dmount4193371320.mount: Succeeded.