setup-python@v2 action overrides LD_LIBRARY_PATH env variable in manylinux container

[gaogaotiantian]

My script works for a long time and suddenly broke recently. Not sure if it's a docker issue or the library env setup.

The problem is similar to https://github.com/pypa/manylinux/issues/357

/usr/bin/docker run --name e5c351c92e44619324a9fb1b8eb57bf888be8_4b2a21 --label 1e5c35 --workdir /github/workspace --rm -e pythonLocation -e LD_LIBRARY_PATH -e INPUT_PYTHON-VERSIONS -e INPUT_BUILD-REQUIREMENTS -e INPUT_SYSTEM-PACKAGES -e INPUT_PRE-BUILD-COMMAND -e INPUT_PACKAGE-PATH -e INPUT_PIP-WHEEL-ARGS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/viztracer/viztracer":"/github/workspace" 1e5c35:1c92e44619324a9fb1b8eb57bf888be8  "cp36-cp36m cp37-cp37m cp38-cp38 cp39-cp39" "" "" "" "" "-w ./dist --no-deps"
+ PY_VERSIONS='cp36-cp36m cp37-cp37m cp38-cp38 cp39-cp39'
+ BUILD_REQUIREMENTS=
+ SYSTEM_PACKAGES=
+ PRE_BUILD_COMMAND=
+ PACKAGE_PATH=
+ PIP_WHEEL_ARGS='-w ./dist --no-deps'
+ cd /github/workspace/
+ '[' '!' -z '' ']'
+ '[' '!' -z '' ']'
+ arrPY_VERSIONS=(${PY_VERSIONS// / })
+ for PY_VER in '"${arrPY_VERSIONS[@]}"'
+ /opt/python/cp36-cp36m/bin/pip install --upgrade --no-cache-dir pip
Requirement already up-to-date: pip in /opt/_internal/cpython-3.6.12/lib/python3.6/site-packages (20.2.3)
+ '[' '!' -z '' ']'
+ /opt/python/cp36-cp36m/bin/pip wheel . -w ./dist --no-deps
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace
Building wheels for collected packages: viztracer
  Building wheel for viztracer (setup.py): started
  Building wheel for viztracer (setup.py): finished with status 'done'
  Created wheel for viztracer: filename=viztracer-0.8.1-cp36-cp36m-linux_x86_64.whl size=714538 sha256=ec8a6197af611f0e464023df7f7d3dafbf290be9477f496ea0239a2e7d2b39c2
  Stored in directory: /tmp/pip-ephem-wheel-cache-f1611rw4/wheels/cb/ab/8f/fd29149f371a05d1292662735ad3569bffe7dbc23f93e70d91
Successfully built viztracer
+ for PY_VER in '"${arrPY_VERSIONS[@]}"'
+ /opt/python/cp37-cp37m/bin/pip install --upgrade --no-cache-dir pip
/opt/_internal/cpython-3.7.9/bin/python3.7: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

[gaogaotiantian]

Adding export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/ in pre-buid-command fixed the issue, but maybe that should be in the project somehow?

[tlestang]

I noticed the same today. In the manylinux image (at least for manylinux2010 which I'm susing), libcrypt.so.2 lives inside /usr/local/lib, which is listed in the LD_LIBRARY_PATH.

However, when running the action, a flag -e LD_LIBRARY_PATH is passed to docker run, which is probably overriding its value in the image... From the workflow logs:

/usr/bin/docker run --name e5c3514cf82e21b34543beaae96ab0ceddf34_767251 --label 1e5c35 --workdir /github/workspace --rm -e pythonLocation -e LD_LIBRARY_PATH -e INPUT_PYTHON-VERSIONS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/PyBaMM/PyBaMM":"/github/workspace" 1e5c35:14cf82e21b34543beaae96ab0ceddf34  "cp36-cp36m cp37-cp37m"

Interestingly, looking at a recent similar workflow (from 19 days ago), the option -e LD_LIBRARY_PATH isn't passed to docker run...

[RalfG]

Strange... I re-ran tests for Python 3.6, 3.7, and 3.8 with manylinux2010, and -e LD_LIBRARY_PATH isn't passed in any of them. One difference with the tests is that they are not using the action directly, but building the container in place. Not sure if this would change anything on this end.

The action doesn't control which environment variables (except for arguments) are sent over to the container.

@gaogaotiantian, @tlestang, could you post a link to the action runs that failed?

[tlestang]

Hi @RalfG In my case the action is also not ran directly, because I needed to customize the Dockerfile. The way wheels are built is similar to the way the action is tested, i.e. by building the docker image in place. Here's a example failed run: https://github.com/tlestang/PyBaMM/runs/1259747579?check_suite_focus=true

[gaogaotiantian]

https://github.com/gaogaotiantian/viztracer/runs/1257990060?check_suite_focus=true

This is one of the action that failed. Hope it would be helpful/

[nijel]

Another failed build is here: https://github.com/WeblateOrg/siphashc/runs/1286221254?check_suite_focus=true

[polm]

I ran into the same issue here:

https://github.com/polm/fugashi/runs/1291199306?check_suite_focus=true

[CaselIT]

I think it's a change in github. At least on my case, this is the docker run command for a successful run 21 days ago:

/usr/bin/docker run --name e5c35e0346b7e32b841e4ac30a46bcea33215_7e959d --label 1e5c35 --workdir /github/workspace --rm -e PYTHONNOUSERSITE -e pythonLocation -e INPUT_PYTHON-VERSIONS -e INPUT_BUILD-REQUIREMENTS -e INPUT_PIP-WHEEL-ARGS -e INPUT_SYSTEM-PACKAGES -e INPUT_PRE-BUILD-COMMAND -e INPUT_PACKAGE-PATH -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/sqlalchemy/sqlalchemy":"/github/workspace" 1e5c35:e0346b7e32b841e4ac30a46bcea33215  "cp27-cp27m" "setuptools>=44 wheel>=0.34" "" "" "" "-w ./dist --no-use-pep517 -v --no-deps"

this is one that has failed now

/usr/bin/docker run --name e5c35eec274791674458c97be80a5715f072b_c0e630 --label 1e5c35 --workdir /github/workspace --rm -e PYTHONNOUSERSITE -e pythonLocation -e LD_LIBRARY_PATH -e INPUT_PYTHON-VERSIONS -e INPUT_BUILD-REQUIREMENTS -e INPUT_PIP-WHEEL-ARGS -e INPUT_SYSTEM-PACKAGES -e INPUT_PRE-BUILD-COMMAND -e INPUT_PACKAGE-PATH -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/sqlalchemy/sqlalchemy":"/github/workspace" 1e5c35:eec274791674458c97be80a5715f072b  "cp27-cp27mu" "setuptools>=44 wheel>=0.34" "" "" "" "-w ./dist --no-use-pep517 -v --no-deps"

They added -e LD_LIBRARY_PATH

[CaselIT]

It think it's the actions/setup-python@v2 that sets it.

Before running it I have this env PYTHONNOUSERSITE: 1, after I have PYTHONNOUSERSITE: 1; pythonLocation: /opt/hostedtoolcache/Python/3.8.6/x64; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.6/x64/lib

[CaselIT]

It think it's the actions/setup-python@v2 that sets it.

Before running it I have this env PYTHONNOUSERSITE: 1, after I have PYTHONNOUSERSITE: 1; pythonLocation: /opt/hostedtoolcache/Python/3.8.6/x64; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.6/x64/lib

Indeed it seems to be this issue https://github.com/actions/setup-python/issues/131

@RalfG I think that the script that runs could be updated so that the correct LD_LIBRARY_PATH is appended to the one provided by the user.

[RalfG]

A workaround has been implemented in #28, now available in RalfG/python-wheels-manylinux-build@v0.3.2