Overwriting `$HOME` causes container to behave differently when executed by runner

[zfields]

Describe the bug On my development machine, when I invoke a container (as root), with a given entrypoint and args, I am seeing a different behavior than when the runner invokes the container with identical entrypoint and args in the cloud with GitHub CI.

The runner invokes the container with the following parameters:

/usr/bin/docker run --name b31692022005eae5ca3fcf20e69bc4e8bc0_682e25 --label 294b31 --workdir /github/workspace --rm -e INPUT_EXAMPLE-SKETCH -e INPUT_FULLY-QUALIFIED-BOARD-NAME -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e GITHUB_STEP_SUMMARY -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true --entrypoint "arduino-cli" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/note-arduino/note-arduino":"/github/workspace" 294b31:692022005eae5ca3fcf20e69bc4e8bc0  "compile" "--build-property" "build.extra_flags=-Werror" "--fqbn" "esp32:esp32:featheresp32" "--warnings" "all" "./examples/Example0_LibrarylessCommunication/Example0_LibrarylessCommunication.ino"

Removing the environment variables we are left with:

/usr/bin/docker run --name b31692022005eae5ca3fcf20e69bc4e8bc0_682e25 --label 294b31 --workdir /github/workspace --rm --entrypoint "arduino-cli" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/note-arduino/note-arduino":"/github/workspace" 294b31:692022005eae5ca3fcf20e69bc4e8bc0  "compile" "--build-property" "build.extra_flags=-Werror" "--fqbn" "esp32:esp32:featheresp32" "--warnings" "all" "./examples/Example0_LibrarylessCommunication/Example0_LibrarylessCommunication.ino"

Removing the volumes (except for the source code) we are left with:

/usr/bin/docker run --name b31692022005eae5ca3fcf20e69bc4e8bc0_682e25 --label 294b31 --workdir /github/workspace --rm --entrypoint "arduino-cli" -v "/home/runner/work/note-arduino/note-arduino":"/github/workspace" 294b31:692022005eae5ca3fcf20e69bc4e8bc0  "compile" "--build-property" "build.extra_flags=-Werror" "--fqbn" "esp32:esp32:featheresp32" "--warnings" "all" "./examples/Example0_LibrarylessCommunication/Example0_LibrarylessCommunication.ino"

Finally remove the name and label parameters and we are left with:

/usr/bin/docker run --workdir /github/workspace --rm --entrypoint "arduino-cli" -v "/home/runner/work/note-arduino/note-arduino":"/github/workspace" 294b31:692022005eae5ca3fcf20e69bc4e8bc0  "compile" "--build-property" "build.extra_flags=-Werror" "--fqbn" "esp32:esp32:featheresp32" "--warnings" "all" "./examples/Example0_LibrarylessCommunication/Example0_LibrarylessCommunication.ino"

On my machine, I invoke the container similarly, like so:

docker run --workdir /host-volume/ --rm --entrypoint "arduino-cli" --volume "$(pwd)":/host-volume/ arduino-buildpack "compile" "--build-property" "build.extra_flags=-Werror" "--fqbn" "esp32:esp32:featheresp32" "--warnings" "all" "./examples/Example0_LibrarylessCommunication/Example0_LibrarylessCommunication.ino"

However, when I invoke the container locally, I get this result

Sketch uses 235953 bytes (18%) of program storage space. Maximum is 1310720 bytes.
Global variables use 16404 bytes (5%) of dynamic memory, leaving 311276 bytes for local variables. Maximum is 327680 bytes.

while the result I receive in GHCI is:

 0 / ?    0.00%
Updating index: library_index.json.gz downloaded

 0 / 543    0.00%
Updating index: library_index.json.sig downloaded

 0 / 495264    0.00%
Updating index: package_index.json downloaded

 0 / 543    0.00%
Updating index: package_index.json.sig downloaded
Downloading missing tool builtin:mdns-discovery@1.0.5...

 0 / 2466817    0.00%
builtin:mdns-discovery@1.0.5 downloaded
Installing builtin:mdns-discovery@1.0.5...
builtin:mdns-discovery@1.0.5 installed
Downloading missing tool builtin:serial-monitor@0.9.1...

 0 / 1954589    0.00%
builtin:serial-monitor@0.9.1 downloaded
Installing builtin:serial-monitor@0.9.1...
builtin:serial-monitor@0.9.1 installed
Downloading missing tool builtin:ctags@5.8-arduino11...

 0 / 111604    0.00%
builtin:ctags@5.8-arduino11 downloaded
Installing builtin:ctags@5.8-arduino11...
builtin:ctags@5.8-arduino11 installed
Downloading missing tool builtin:serial-discovery@1.3.2...

 0 / 1630523    0.00%
builtin:serial-discovery@1.3.2 downloaded
Installing builtin:serial-discovery@1.3.2...
builtin:serial-discovery@1.3.2 installed
Error during build: Platform 'esp32:esp32' not found: platform not installed
Platform esp32:esp32 is not found in any known index
Maybe you need to add a 3rd party URL?

It is behaving as if the file system is messed up. The image installs the supporting files for the Arduino CLI in /root, and the volumes mounted by GHCI are limited to /var and /github so they should not be interfering with each other.

What else is happening that would make this behave differently?

To Reproduce Steps to reproduce the behavior:

1. Compile the following Dockerfile into an image

   # Copyright 2022 Blues Inc.  All rights reserved.
   # Use of this source code is governed by licenses granted by the
   # copyright holder including that found in the LICENSE file.
   
   # Build development environment
   # docker build --file Dockerfile.arduino-cli --tag arduino-buildpack .
   
   # Launch development environment
   # docker run --entrypoint bash --interactive --rm --tty --volume "$(pwd)":/host-volume/ --workdir /host-volume/ arduino-buildpack
   
   # Define global arguments
   ARG DEBIAN_FRONTEND="noninteractive"
   ARG UID=1000
   ARG USER="blues"
   
   # POSIX compatible (Linux/Unix) base image
   FROM debian:stable-slim
   
   # Import global arguments
   ARG DEBIAN_FRONTEND
   
   # Define local arguments
   ARG ARDUINO_CLI_VERSION=0.21.1
   ARG BINDIR=/usr/local/bin
   ARG ECHO_BC_FILE='$bcfile'
   
   # Define environment variables
   ENV ARDUINO_UPDATER_ENABLE_NOTIFICATION=false
   
   # Establish development environment
   RUN ["dash", "-c", "\
      apt-get update --quiet \
   && apt-get install --assume-yes --no-install-recommends --quiet \
       bash \
       ca-certificates \
       curl \
       python-is-python3 \
       python3 \
       python3-pip \
       ssh \
   && pip install \
       pyserial \
   && apt-get clean \
   && apt-get purge \
   && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
   "]
   
   # Download/Install Arduino CLI
   RUN ["dash", "-c", "\
      curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | BINDIR=${BINDIR} sh -s ${ARDUINO_CLI_VERSION} \
   # && arduino-cli completion bash > /usr/share/bash-completion/completions/arduino-cli.sh \
   && mkdir -p /etc/bash_completion.d/ \
   && arduino-cli completion bash > /etc/bash_completion.d/arduino-cli.sh \
   && echo >> /etc/bash.bashrc \
   # && echo \"for bcfile in /usr/share/bash-completion/completions/* ; do\" >> /etc/bash.bashrc \
   #  && echo \"for bcfile in /etc/bash_completion.d/* ; do\" >> /etc/bash.bashrc \
   #  && echo \"    [ -f \\\"${ECHO_BC_FILE}\\\" ] && . \\\"${ECHO_BC_FILE}\\\"\" >> /etc/bash.bashrc \
   #  && echo \"done\" >> /etc/bash.bashrc \
   "]
   
   # Configure Arduino CLI
   RUN ["dash", "-c", "\
      arduino-cli config init \
   && arduino-cli config add board_manager.additional_urls \
       https://raw.githubusercontent.com/stm32duino/BoardManagerFiles/main/package_stmicroelectronics_index.json \
       https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json \
       https://raw.githubusercontent.com/sparkfun/Arduino_Apollo3/main/package_sparkfun_apollo3_index.json \
       https://raw.githubusercontent.com/adafruit/arduino-board-index/gh-pages/package_adafruit_index.json \
       https://github.com/earlephilhower/arduino-pico/releases/download/global/package_rp2040_index.json \
   && arduino-cli core update-index \
   && arduino-cli core install arduino:avr \
   && arduino-cli core install arduino:mbed_nano \
   && arduino-cli core install arduino:samd \
   && arduino-cli core install STMicroelectronics:stm32 \
   && arduino-cli core install esp32:esp32 \
   && arduino-cli core install SparkFun:apollo3 \
   && arduino-cli core install adafruit:nrf52 \
   && arduino-cli core install adafruit:samd \
   && arduino-cli core install rp2040:rp2040 \
   && arduino-cli lib install \"Blues Wireless Notecard\" \
   && arduino-cli lib install \"Adafruit BME680 Library\" \
   "]
   
   ENTRYPOINT ["arduino-cli"]
   
   CMD ["help"]

2. Attempt to compile an Arduino .ino file

3. See error

Expected behavior Given the same inputs, I would expect the Docker container to produce the same outputs both locally and in the cloud with GHCI.

Runner Version and Platform

Version of your runner?

OS of the machine running the runner? ubuntu-latest

What's not working?

Please include error messages and screenshots.

Job Log Output

If applicable, include the relevant part of the job / step log output here. All sensitive information should already be masked out, but please double-check before pasting here.

Runner and Worker's Diagnostic Logs

If applicable, add relevant diagnostic log information. Logs are located in the runner's _diag folder. The runner logs are prefixed with Runner_ and the worker logs are prefixed with Worker_. Each job run correlates to a worker log. All sensitive information should already be masked out, but please double-check before pasting here.

[zfields]

I figured it out. You are passing the parameter -e HOME to the docker run command. In doing so, you are setting the HOME environment variable to /github/home/, instead of leaving it as /root which is the root user's home directory.

This causes me a problem, because I install tools in my container and the tools subsequently create subfolders under the user's home directory. When the runner changes the meaning of $HOME, then my installed tools can no longer identify where their directories are installed.

As a workaround, I am using bash as the entrypoint and using an inline environment variable declaration to restore HOME to the expected value. This is undesirable, because I can no longer use the ENTRYPOINT keyword in the docker file or the action YAML for the utility I wish to run.

Workaround:

bash -c "HOME=/root arduino-cli compile --build-property build.extra_flags=-Werror --fqbn arduino:avr:uno --warnings all sketch.ino"

If updating the HOME environment variable is not critical to the functionality of GitHub CI, please consider removing it from the docker run command. :pray:

[sidwarkd]

+1 for not overwriting the $HOME folder as container behavior may rely on it. Would be better if Github had their own $GITHUB_HOME.

[zfields]

I did some digging, and HOME makes the list of RESERVED variables in some texts... https://www.linuxtopia.org/online_books/introduction_to_linux/linux_Reserved_variables.html

While in others, it is heavily frowned upon... https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html

It is unwise to conflict with certain variables that are frequently exported by widely used command interpreters and applications:

If nothing else, this should be a heavy indicator of potential name collision and impact to containerized applications.

[jj51726-jd]

environment variables are pretty much equivalent to global variables and modification should be avoided for that reason. they should also not be relied upon to have a specific or known-good value, because as you've discovered, they can be modified or relied upon by anything. call your executables via their absolute path, and use absolute paths everywhere you use relative paths or rely on the $PATH environment variable today.

[zfields]

@jj51726-jd nothing you said is wrong, but I'm afraid it misses the mark in this case. That being said, it does indicate that I was unclear in my writing above, and this is an opportunity to clarify the original text.

I'll restate it here, in case it helps to clarify the problem for others...

During image creation, I install certain tools through my Dockerfile. Then, during a subsequent step of the image creation, I instruct one of the newly installed tools to install it's own dependencies. The tool uses ${HOME} to identify where it should install tools - without prompting or taking input from me.

After container instantiation, the tool then looks back to ${HOME} to find it's dependencies. However, the GitHub Action Runner has overridden ${HOME}, and now the tool thinks it's dependencies are not installed.

In this instance, I have no option to supply paths of any kind, everything is internal to the 3rd-party tool. Furthermore, this is a fairly common pattern, so we can't just blame this one tool provider and expect them to change it and fix everything. It would be easier, for the GitHub Action Runner to create it's own HOME variable (like GITHUB_HOME as @sidwarkd suggested), because the runner would be aware of it and could use it as necessary.

As a result, my workaround was to provide the original value of ${HOME} to the tool, so it would be able to locate the dependencies it installed during image creation.

[arg0d]

This is an awful bug in Github actions. Who thought its OK to overwrite $HOME variable? Overwriting $HOME variable breaks a lot of intuitive assumptions about Linux environment. What is the reason for overwriting $HOME variable? If overwriting $HOME variable does not benefit the users directly, this needs to be fixed ASAP.