mahabaleshwars
commented
3 months ago Hello @nvincent-vossloh, Thank you for creating this issue. We will investigate it and provide feedback as soon as we have some updates.
Open nvincent-vossloh opened 3 months ago
mahabaleshwars
commented
3 months ago Hello @nvincent-vossloh, Thank you for creating this issue. We will investigate it and provide feedback as soon as we have some updates.
aparnajyothi-y
commented
3 months ago Hello @nvincent-vossloh, Thank you once again for creating this issue. We have investigated and found that the problem is due to environment variables not being available in the context where the action is running. The given logs indicate the inclusion of a path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which does not exist on the self-hosted runner. You need to ensure that the actions/setup-python action respects the custom tool cache directory by setting the environment variables directly in the workflow, as shown in the job below:
jobs:
test-python:
name: Test Python pip install
runs-on: [self-hosted, linux, test-python]
env:
RUNNER_TOOL_CACHE: ${{ github.workspace }}/_work/_tool
AGENT_TOOLSDIRECTORY: ${{ github.workspace }}/_work/_tool
steps:
- name: Github Actions Context Information
uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Show environment variables
run: env
- name: Python package install
run: pip install PyGObjectyou can use any custom directory to install python on your self-hosted runner using AGENT_TOOLSDIRECTORY environment variable as per the setup-python documentation. Pip gets the include path from the environment variables set up during the installation of Python. Specifically, the paths such as PKG_CONFIG_PATH, Python_ROOT_DIR, Python2_ROOT_DIR, and Python3_ROOT_DIR are configured to point to the respective directories during the setup process.
aparnajyothi-y
commented
2 months ago Hello @nvincent-vossloh, Please let us know in case of any further concerns on the above information.
nvincent-vossloh
commented
2 months ago Thanks for your feedback, I will try your solution and let you know if that fixed the issue on my side.
nvincent-vossloh
commented
2 months ago The documentation you link to, states the the environment variable could be set in the same shell where the runner is running. Unfortunately that does not work (anymore ?), that is why I opened the issue in the first place. Maybe the runner software does not use the env variable from the shell anymore (did not work with the .env file either).
Anyway, unfortunately setting the env variable in the workflow does not fix the issue, the behavior is still the same. On my first try it succeeded, but that was because the pip install step used a cache version of PyGObject, after deleting it the failure to compile still happens.
Only setting CPATH seems to work.
gowridurgad
commented
1 month ago Hi @nvincent-vossloh, After further investigation, we found that setting the environment variable in the workflow successfully works when the cached version of PyGObject is available. However, after clearing the cache, the compilation failure reoccurs. Given these circumstances, It appears that only setting CPATH directly works. This is because CPATH explicitly sets the include path for the compiler, ensuring the correct headers are found during the compilation of PyGObject. The default paths set by pip or setup-python may not align with the directory structure on the self-hosted runner, leading to the observed compilation errors.
Setting CPATH ensures that the compiler can locate the necessary Python header files, avoiding the "No such file or directory" error for Python.h. This approach provides a direct and reliable way to specify include directories, bypassing any inconsistencies in environment variable handling between the workflow and the self-hosted runner.
Thank you for your understanding and patience as we work through this issue.
nvincent-vossloh
commented
1 month ago Do you plan on updating the setup-python action in order to add the include directory to the CPATH env var ?
gowridurgad
commented
1 month ago Hi @nvincent-vossloh, Thank you for suggesting to include directory to the CPATH environment variable in the setup-python action. We acknowledge the importance of this feature and will explore the possible ways to implement it. We will consider this for a future enhancement.
In the meantime, here is an alternative solution to include CPATH in a GitHub Actions workflow: Modify the workflow to set the CPATH environment variable: The CPATH environment variable can be set directly in the workflow file using the env key.
jobs:
test-python:
name: Test Python pip install
runs-on: self-hosted
steps:
- name: Github Actions Context Information
uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Set CPATH environment variable
run: echo "CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10" >> $GITHUB_ENV
- name: Python package install
run: pip install PyGObjectHi @nvincent-vossloh, Thank you for suggesting to include directory to the CPATH environment variable in the setup-python action. We acknowledge the importance of this feature and will explore the possible ways to implement it. We will consider this for a future enhancement.
Thank you for taking my suggestion into account.
In the meantime, here is an alternative solution to include CPATH in a GitHub Actions workflow: Modify the workflow to set the CPATH environment variable: The CPATH environment variable can be set directly in the workflow file using the env key.
jobs: test-python: name: Test Python pip install runs-on: self-hosted steps: - name: Github Actions Context Information uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7 - uses: actions/setup-python@v5 with: python-version: '3.10' - name: Set CPATH environment variable run: echo "CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10" >> $GITHUB_ENV - name: Python package install run: pip install PyGObject
In the end I used this:
env:
pytest_python_version: '3.10'
[...]
- uses: actions/setup-python@v5
id: setup-python
with:
python-version: ${{env.pytest_python_version}}
- name: setup CPATH
run: |
echo "python version : ${{ steps.setup-python.outputs.python-version }}"
python_version="${{ env.pytest_python_version }}"
echo "python path : ${{ steps.setup-python.outputs.python-path }}"
CPATH=${pythonLocation}/include/python${python_version}
echo CPATH=${CPATH} >> "${GITHUB_ENV}"which leverage the various env variables set.
Thanks again.
Description: I have exported
RUNNER_TOOL_CACHEandAGENT_TOOLSDIRECTORYin the shell running the self-hosted runner, however apip installin the workflow still uses/opt/hostedtoolcachedirectory. job-log.txtAction version:
actions/setup-python@v5Platform:
Runner type:
Tools version: python-version: '3.10'
Repro steps:
From the Self-hosted runner directory:
Expected behavior: I would expect the
pip installstep to set properly the-Iflags without/opt/hostedtoolcacheActual behavior: The logs (see above) show that it tried to add an include path (
-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which do not exist on the self hosted runner.Workaround I am not fond of
With this workaround I manage to install the lib, however it looks like that pip cached some stuff in the default cache directory:
(
Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97) even though theRUNNER_TOOL_CACHEandAGENT_TOOLSDIRECTORYhave been set up.I took the idea of this workaround from https://github.com/actions/setup-python/issues/489 and https://github.com/actions/setup-python/issues/485 However I do not understand why the
RUNNER_TOOL_CACHEandAGENT_TOOLSDIRECTORYare not used.Beside, if i run
from
_work/_toolI get a lot of match:I don't really know where pip gets the include path from, but I guess from one of the above files which clearly points to an invalid folder.
Thanks in advance for your help, let me know if you need more information.