Closed SirensOfTitan closed 9 months ago
Hey @SirensOfTitan,
Can you please describe your setup? Based on the log, I assume you are running runner in k8s cluster inside the docker image and using docker hook. If that is the case, can you check if the user inside the job container has the same UID as the runner user?
This permission denied can happen in case your runner user is running as UID 1001, and you have a user inside the job container having UID different from 1001. Then, since /__t
is mounted from the runner, it inherits the permission on the host. So either your job container user needs to be root
, or it needs to match the UID of the runner user.
Hi @nikola-jokic : Thank you for the speedy response!
Can you please describe your setup? Based on the log, I assume you are running runner in k8s cluster inside the docker image and using docker hook. If that is the case, can you check if the user inside the job container has the same UID as the runner user?
Yes, that's correct. I'm using an image based on ghcr.io/actions/actions-runner-controller/actions-runner:ubuntu-22.04
(source: https://github.com/actions/actions-runner-controller/blob/master/runner/actions-runner.ubuntu-22.04.dockerfile) with runner container hooks updated to 0.3.1 (that original one is set to 0.2.0 by default) as the container for the child pod in question and the default runner container image.
... as such, the runner has UID 1001 in both cases. In my case, the __t
directory doesn't seem to exist at all in the child pod:
cd /t bash: cd: /t: No such file or directory
I'll check the volume mounts and get back to you as soon as I can :relaxed:
Hey @SirensOfTitan,
I think I found the issue. Can you please share how are you using setup-python (the step and with
parameters)? I investigated differences between hook and the runner suspecting there is a mount that the runner did not send. Since the container hook respects mounts sent from the runner, this can't be an issue related to the container hook.
Anyway, the check if the hostedtoolcache exists tells me that you are probably using caching. This mount is added by default on hosted runners (see setup-python docs), but the __t
mount does not exist on the self-hosted runner. Then, the execution probably tries to create /__t
but since it is not a root, it does not have permissions to create directories in the /
path causing permission issue. :smile: Let me know if that is actually the issue. I tried comparing docker execution on self-hosted runner without the hook, and it does not come with the /__t
mount as well
Here is the similar issue in the runner: https://github.com/actions/runner/issues/2522
Same problem with /opt/hostedtoolcache
regardless of if the hook is being used :relaxed:
@nikola-jokic: Thank you!
Here's the setup-python step:
- uses: actions/setup-python@v4
with:
cache: "poetry"
cache-dependency-path: "./poetry.lock"
We are indeed using caching! If it helps: we're on Enterprise Server 3.8 as well.
This mount is added by default on hosted runners (see setup-python docs), but the t mount does not exist on the self-hosted runner. Then, the execution probably tries to create /t but since it is not a root, it does not have permissions to create directories in the / path causing permission issue.
This is correct! The /t mount doesn't exist like e.g. the work folder at /w, so the execution tries to create and fails since it isn't root.
I'll leave this issue open here, but until the fix is ready in the runner. :relaxed: Thank you for reporting it!
Let's close this issue here since it should be fixed on the runner. Thank you again for reporting it!
When running something that relies on hostedtoolcache as a non-root user, like
setup-python
, I run into the following issue:The work directory is there and looks alright at
__w
, there is also a__e
directory present that seems pertinent.