Enabling persistence of home directory breaks UDI home directory, PATH and environment variables

[AObuchow]

Describe the bug

https://github.com/eclipse/che/issues/22102 added support for automatically mounting the /home/user/ directory to a PVC. However, in the UDI, the /home/user/ directory has configurations and tools that are overwritten when spec.devEnvironments.persistUserHome.enabled: true is set in the Che Cluster CR. This is caused by the PVC's /home/user/ directory being mounted overwriting the UDI's /home/user/ directory.

Many environment variables in the UDI point to /home/user/, bash completions are stored in /home/user/.bashrc. This also affects $PATH, which references /home/user/.

For example, grepping /home/user/ in the UDI dockerfile:

[aobuchow@fedora ubi8]$ cat Dockerfile | grep "/home/user/"
    && bash -c ". /home/user/.sdkman/bin/sdkman-init.sh \
             && sed -i "s/sdkman_auto_answer=false/sdkman_auto_answer=true/g" /home/user/.sdkman/etc/config \
             && sed -i "s/sdkman_auto_env=false/sdkman_auto_env=true/g" /home/user/.sdkman/etc/config \
ENV JAVA_HOME_8=/home/user/.sdkman/candidates/java/8.0.332-tem
ENV JAVA_HOME_11=/home/user/.sdkman/candidates/java/11.0.15-tem
ENV JAVA_HOME_17=/home/user/.sdkman/candidates/java/17.0.3-tem
# Java-related environment variables are described and set by /home/user/.bashrc
ENV SDKMAN_CANDIDATES_DIR="/home/user/.sdkman/candidates"
ENV SDKMAN_DIR="/home/user/.sdkman"
ENV GRADLE_HOME="/home/user/.sdkman/candidates/gradle/current"
ENV JAVA_HOME="/home/user/.sdkman/candidates/java/current"
ENV MAVEN_HOME="/home/user/.sdkman/candidates/maven/current"
ENV GRAALVM_HOME=/home/user/.sdkman/candidates/java/22.1.0.0.r17-mandrel
ENV PATH="/home/user/.krew/bin:$PATH"
ENV PATH="/home/user/.sdkman/candidates/maven/current/bin:$PATH"
ENV PATH="/home/user/.sdkman/candidates/java/current/bin:$PATH"
ENV PATH="/home/user/.sdkman/candidates/gradle/current/bin:$PATH"
ENV PATH="/home/user/.local/share/coursier/bin:$PATH"
ENV NVM_DIR="/home/user/.nvm"
RUN source /home/user/.bashrc && nvm install v${NODEJS_VERSION} && nvm install v${NODEJS_14_VERSION} && nvm install v${NODEJS_12_VERSION} && nvm alias default v$NODEJS_VERSION && nvm use v$NODEJS_VERSION && npm install --global yarn@v1.22.17
ENV KUBECONFIG=/home/user/.kube/config
# gopls 0.10+ - installed to /home/user/go/bin/gopls and /home/user/go/pkg/mod/
ENV GOBIN="/home/user/go/bin/"
ENV CARGO_HOME=/home/user/.cargo \
    RUSTUP_HOME=/home/user/.rustup \
    PATH=/home/user/.cargo/bin:${PATH}
RUN echo "source /usr/share/bash-completion/completions/git" >> /home/user/.bashrc
    && echo "source /usr/share/bash-completion/completions/oc" >> /home/user/.bashrc
echo '[ -f ~/.kubectl_aliases ] && source ~/.kubectl_aliases' >> /home/user/.bashrc
echo "source /usr/share/bash-completion/completions/kubectl" >> /home/user/.bashrc
echo 'export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"' >> /home/user/.bashrc

Grepping /home/user/ in $PATH:

projects $ echo $PATH | grep "/home/user/"
/home/user/.krew/bin:/home/user/.sdkman/candidates/jbang/current/bin:/home/user/.local/bin:/home/user/bin:/usr/share/Modules/bin:/home/user/.cargo/bin:/home/user/go/bin/:/home/user/.nvm/versions/node/v16.14.0/bin:/home/user/.local/share/coursier/bin:/home/user/.sdkman/candidates/gradle/current/bin:/home/user/.sdkman/candidates/java/current/bin:/home/user/.sdkman/candidates/maven/current/bin:/home/user/.krew/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.dotnet/tools

Che version

7.72@latest

Steps to reproduce

1. Set spec.devEnvironments.persistUserHome.enabled: true in the CheCluster CR
2. Try running the mvn command, it will show that it could not be found as it is no longer in the expected location on $PATH

Expected behavior

All environment variables and commands that were configured to use /home/user/ should work as expected when spec.devEnvironments.persistUserHome.enabled: true in the CheCluster CR

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

Linux

Eclipse Che Logs

No response

Additional context

No response

[AObuchow]

I wonder if we could just move all configuration in the UDI to point to /home/tooling/ instead of /home/user/?

Edit: I tested this approach, and it seems to work so far, though there might be some issues I haven't seen yet. I made quay.io/aobuchow/universal-developer-image:tooling-dir if anyone wants to try it out, though I forgot to set $HOME back to /home/user/ in that image.

[AObuchow]

So far, my proposed solution for this issue is as follows:

1. Move all tool installations from /home/user/* to /home/tooling/.
   • This requires setting $HOME to /home/tooling/ during the installation of packages, .bashrc modifications, etc and then setting $HOME back to /home/user/ afterwards, so that $HOME remains unchanged when the container is in use.
2. Use GNU stow to create symlinks for everything in /home/tooling/ (located in the pod's ephemeral storage) to point to /home/user/ (which is located on the PVC). Symbolic links should work better than directly copying between /home/tooling/ and /home/user/, as it will allow for easily swapping out the contents of /home/tooling/ when the UDI is updated. If we were to copy, we'd have deleted the outdated contents on the PVC first before re-copying the update UDI contents (and face other complications)

Implementing 2. requires a post-start event on the workspace pod to invoke stow after the PVC has been mounted (otherwise any symbolic links created by stow will be overwritten by the PVC's /home/user/ directory).

I don't think it makes sense to have DWO insert this specific post-start event, as the post-start event relies on the existence of /home/tooling and stow being installed on the workspace container image. Overall, the issue at hand (and my proposed solution) is more specific to Che and the UDI rather than DWO itself.

Thus, the User Dashboard could potentially insert this post-start event into devworkspaces.

Alternatively, DWO + Che-Operator could insert this post start event in a more generic way:

• Allow for default post-start events in DWO that are automatically applied to all workspaces. This could be configured in the DWOC, something like config.workspace.events.default.postStart
• Have Che-Operator configure the DWOC's default post-start event to invoke stow

Any thoughts on the overall proposed solution, and whether we should take a User Dashboard or Che-Operator + DWO approach? @amisevsk @l0rd @ibuziuk

[amisevsk]

It might make sense to include the stow step in the entrypoint.sh of the UDI image, if we're taking this approach.

[AObuchow]

It might make sense to include the stow step in the entrypoint.sh of the UDI image, if we're taking this approach.

This would be the nicest solution, however I couldn't get it to work.

Adding the stow step to the entrypoint worked when running the UDI with docker as a container in isolation, but it didn't seem to work in a che-code workspace (which is weird, because we moved the Che-Code entrypoint logic to a post-start command, and kubedock depends the entrypoint already).

I might have made a mistake somewhere in my attempt, though.

[AObuchow]

It might make sense to include the stow step in the entrypoint.sh of the UDI image, if we're taking this approach.

This would be the nicest solution, however I couldn't get it to work.

Adding the stow step to the entrypoint worked when running the UDI with docker as a container in isolation, but it didn't seem to work in a che-code workspace (which is weird, because we moved the Che-Code entrypoint logic to a post-start command, and kubedock depends the entrypoint already).

I might have made a mistake somewhere in my attempt, though.

:facepalm: My devfile had was overwriting the entrypoint of the container. @amisevsk's suggested solution of using the UDI's entrypoint to run the stow step works as expected.

[achdmbp]

would it be feasible to copy /home/user content from udi image to the mounted volume when the workspace starts the first time ?

[amisevsk]

would it be feasible to copy /home/user content from udi image to the mounted volume when the workspace starts the first time ?

It's potentially feasible, but the current UDI container stores quite a bit of data in /home/user. Copying this data into the persistent volume might cause other issues (increasing time before workspace is ready, consuming ~2 GB of data in the PV needlessly).

[achdmbp]

what is the resolution on this for users that use their own custom image instead of the udi?