home directory not writable

[xenon-dev]

First of all, thanks for the slim and awesome image!

We are using it inside a kubernetes cluster hosted on IBM Cloud.

After switching from a CI pipeline within GitLab.com to a selfhosted ArgoWorkflow Pipeline, we are not able to get the pod started after building the image. With the old pipeline it worked flawlessly for many months.

We get exactly the following error that you already mentioned in closed issue#4, but we are not using bindmounts or persistent volumes for the /home/headless/ path.

The error comes into existence (we stripped our Dockerfile.txt clean, to see, when it occurs), with the line 49 (see attached Dockerfile): COPY ./config/images/ ./config/startup.sh /home/headless/

Afterwards we also set the execution rights: RUN chown -R 1001:1000 /home/headless/startup.sh and: chmod u+x /home/headless/startup.sh

The actual execution is inside the stateful-set (like a deployment) and was commented out. So even withouth execution, the pod will fail. The script itself is only used to copy a file and set the log path.

Details and environment (Dockerfile.txt excerpts attached):

• Release 22.04 was used in all tests
• Kubernetes Version 1.22.6
• We build with Kaniko instead of DinD for security reasons and caching is switched off, snapshotMode ist set to "full" (we strive for debugging purposes to get a reproducible, clean build)
• The build step is succesfull with no error
• In the old GitLab.com CI/CD pipeline we also build with Kaniko, without any further flags (worked great)
• When starting the pod it fails with:

mkdir: cannot create directory '/home/headless/.vnc': Permission denied /dockerstartup/vnc_startup.rc: line 36: /home/headless/.vnc/passwd: No such file or directory chmod: cannot access '/home/headless/.vnc/passwd': No such file or directory /dockerstartup/vnc_startup.rc: line 43: /home/headless/.vnc/config: No such file or directory and "crashloopbackoff"

• Running $id yields: uid=1001(headless) gid=1000(headless) groups=1000(headless),27(sudo) looks correct, right?
• "Fun"-fact: If the image is build with docker cli locally (on a mac, if that matters), then pushed to the registry and deployed with ArgoCD it works, the image is smaller, has a different python version (3.8) installed (caching was also off for that test).

Any ideas or discussions are helpful, we can happily provide more details if asked for. Thanks for the fine work!

[accetto]

Hello @xenon-dev,

thanks for your feedback. This could be one of those problems, that are not fast to solve. This permission stuff was always nasty.

Recently I've found, for example, that if I start some containers from the Docker Desktop application on Windows, then I'm not able to access some folders. However, that is not the case if I start the same containers from CLI. And the issue #25 with sudo on okteto is also still opened. :)

By the way, I also prefer to build the images locally and then to push them up for some time already. The cloud builders seem to use older stuff sometimes. They also often modify the images for security reasons. However, local building does not work in many scenarios, of course.

Is there any way I could test it on IBM cloud for free? Because I'm afraid that this thing is dependent on the environment.

Regards, accetto

[xenon-dev]

Hey, thanks for answering, we have also read the issue #25, but can´t get to the bottom of the actual problem currently, as we have not changed the image itself, nor the executing environment on the cluster (presumably).

For IBM (and almost every other big cloud player), you can get a free tier (200$) test account: https://www.ibm.com/cloud/free (I don´t know yet what would happen after the 200$ expire, so use with your own care to avoid costs)

We are not running on code-engine, but our own k8s-cluster that belongs solely to us.

(Here is a list for other clouds aswell: https://github.com/learnk8s/free-kubernetes)

[xenon-dev]

Hey! We were able to finally build the image "the right way" with Buildkit. Maybe just a friendly hint: If your users @accetto moan about the image not working in a CI, point them to here: buildkit-template.yaml. IBM explicitly installs a AppArmor Deamonset when initialising the k8s-clusters, only with this addition to the proposed workflow template it worked, before that, we got issues, that the buildkit deamon could not be reached (because it could not be started in the first place):

      metadata:
        annotations:
          container.apparmor.security.beta.kubernetes.io/main: unconfined
      container:
        name: ''
        image: 'moby/buildkit:v0.10.3-rootless'
        command:
          - buildctl-daemonless.sh

Switch to unconfined on pod-basis, instead of cluster-basis to have a little bit of security left..

I will close this issue, because it has really nothing to do with the image itself or our chown commands after building the base image, but with the circumstances, how the cloud provider implement certain build tools. There are a few dozens issues open on Kanikos Repo, with users complaining about mkdir not working inside a run command etc. etc., we chimed in here: Kaniko Issues

Digging down to the root cause and getting the order of those steps right took a good amount of time...

Have fun!

[accetto]

I appreciate your Feedback and I'm glad, that you've found the solution. Good luck!