[BUG]: Container job won't work when agent run in a container

[rballard-ghd]

What happened?

When trying to use container jobs on a self hosted agent, I had the following error when trying to start the container (it pulled the image down correctly):

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/__a/externals/node/bin/node": stat /__a/externals/node/bin/node: no such file or directory: unknown

I believe the issue is that the agent tried to mount /a and /w into the container, however due to the agent running in the container, the docker engine tries to use the host's file system to mount from, and they don't exist.

What should happen

After my investigation, I think containers should be able to provide their own version of node (as outlined for non-glibc containers), and the work directory can be checked out within the container separately. So the container requirements would be node.js and git.

Versions

Agent Version 3.236.1; Linux x64; Ubuntu 22.04

Environment type (Please select at least one enviroment where you face this issue)

• [X] Self-Hosted
• [ ] Microsoft Hosted
• [ ] VMSS Pool
• [X] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22.02

Version controll system

No response

Relevant log output

/usr/bin/docker network create --label c13cac vsts_network_4ea1a74e2e734e69a5540f28ec13f6b3
198b412fbdff319df7067344caf8876b6a1ca20d93ec1285eeebee10f921daf4
/usr/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" docker.io/greenbone/doxygen:latest
/usr/bin/docker create --name ac5d2fcf331a466a910ec565ff3b1184_dockeriogreenbonedoxygenlatest_fc980d --label c13cac --network vsts_network_4ea1a74e2e734e69a5540f28ec13f6b3  -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/__a/_work/1":"/__w/1" -v "/__a/_work/_temp":"/__w/_temp" -v "/__a/_work/_tasks":"/__w/_tasks" -v "/__a/_work/_tool":"/__t" -v "/__a/externals":"/__a/externals":ro -v "/__a/_work/.taskkey":"/__w/.taskkey" docker.io/greenbone/doxygen:latest "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
DockerActionRetries variable value: True
/usr/bin/docker start d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/__a/externals/node/bin/node": stat /__a/externals/node/bin/node: no such file or directory: unknown
Error: failed to start containers: d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
##[warning]Docker start failed with exit code 1, back off 00:00:10 seconds before retry.
/usr/bin/docker start d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/__a/externals/node/bin/node": stat /__a/externals/node/bin/node: no such file or directory: unknown
Error: failed to start containers: d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
##[warning]Docker start failed with exit code 1, back off 00:00:10 seconds before retry.
/usr/bin/docker start d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/__a/externals/node/bin/node": stat /__a/externals/node/bin/node: no such file or directory: unknown
Error: failed to start containers: d88f958d3662b505d912e752f26508189d1fd6d0d1e0b939d7e4d33cf68c3240
##[warning]Docker start failed with exit code 1, back off 00:00:10 seconds before retry.
##[error]Docker start fail with exit code 1

[vmapetr]

Hi @rballard-ghd thanks for reporting! We are working on more prioritized issues at the moment, but will get back to this one soon.

[cjproud]

Bump, I'm seeing this as well when trying to run containers within an azp-agent container.

[hansbonini]

Same here, addressed a PR which fix the issue

[github-actions[bot]]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[rballard-ghd]

Bumping this as this is still a bug/feature request.