search

microsoft / azure-pipelines-agent

Azure Pipelines Agent 🚀
MIT License
1.72k stars 866 forks source link

[BUG]: Self-hosted agent fails to mount whole directory in container if previous job on same agent checks out repository with a path #4901

Open MaximilianGuttmann opened 3 months ago

MaximilianGuttmann commented 3 months ago

What happened?

Hey, I have observed a bug where the my self hosted agent mounts only the source directory /opt/azagent/_work/1/s into the container instead of the pipeline workspace /opt/azagent/_work/1/. This leads to the Default Working Directory being /opt/azagent/_work/1/s/checkedOutRepo.

The problem only occurs if you have 2 jobs with the first one checking out the repository into a specific path and the second job running on the same agent as the first job ran. To ensure that the second job runs on the same agent I divided both jobs into seperate stages. I also created a simple pipeline that reproduces this bug:

variables:
  - name: System.Debug
    value: true

stages:
  - stage: A
    displayName: "Stage A"

    pool: "Debian12_docker"

    jobs:
      - job: A1
        displayName: "Job A1"

        steps:
          - checkout: self
            path: "s/$(Build.Repository.Name)"

  - stage: B
    displayName: "Stage B"
    dependsOn: A

    pool: "Debian12_docker"

    jobs:
      - job: B
        displayName: "Job B"
        container:
          image: "debian:bullseye"
        strategy:
          matrix:
            A:
              name: "A"
            B:
              name: "B"

        steps:
          - checkout: self

The bug does not occur if the second job is executed on another agent. So in this example only one job of the second stage mounts the directory in the wrong place.

There was already such an issue in the past though I don't think that it is not exactly the same. https://github.com/microsoft/azure-pipelines-agent/issues/4479

Versions

Agent Version: 3.242.0 Agent OS: Linux

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

Azure DevOps Server 2022.1

Operation system

No response

Version controll system

git

Relevant log output

2024-07-19T09:24:24.2271790Z ##[debug]Evaluating condition for step: 'Initialize containers'
2024-07-19T09:24:24.2293955Z ##[debug]Evaluating: SucceededNode()
2024-07-19T09:24:24.2297723Z ##[debug]Evaluating SucceededNode:
2024-07-19T09:24:24.2322714Z ##[debug]=> True
2024-07-19T09:24:24.2328102Z ##[debug]Result: True
2024-07-19T09:24:24.2369099Z ##[section]Starting: Initialize containers
2024-07-19T09:24:24.2463364Z DockerActionRetries variable value: False
2024-07-19T09:24:24.2471420Z ##[command]/usr/bin/docker version --format '{{.Server.APIVersion}}'
2024-07-19T09:24:24.2679556Z '1.46'
2024-07-19T09:24:24.2708737Z Docker daemon API version: '1.46'
2024-07-19T09:24:24.2708844Z ##[command]/usr/bin/docker version --format '{{.Client.APIVersion}}'
2024-07-19T09:24:24.2893079Z '1.46'
2024-07-19T09:24:24.2913477Z Docker client API version: '1.46'
2024-07-19T09:24:24.2918663Z ##[debug]Delete stale containers from previous jobs
2024-07-19T09:24:24.2922738Z ##[command]/usr/bin/docker ps --all --quiet --no-trunc --filter "label=dc4b27"
2024-07-19T09:24:24.3069230Z ##[debug]Delete stale container networks from previous jobs
2024-07-19T09:24:24.3089256Z ##[command]/usr/bin/docker network prune --force --filter "label=dc4b27"
2024-07-19T09:24:24.3274332Z DockerActionRetries variable value: False
2024-07-19T09:24:24.3278566Z ##[command]/usr/bin/docker pull debian:bullseye
2024-07-19T09:24:25.5658556Z bullseye: Pulling from library/debian
2024-07-19T09:24:25.5723979Z Digest: sha256:b6b4a479a3bc1b8dfda5544f00b72aafb230bb835a43974a147b220c38df882e
2024-07-19T09:24:25.5724098Z Status: Image is up to date for debian:bullseye
2024-07-19T09:24:25.5731838Z docker.io/library/debian:bullseye
2024-07-19T09:24:25.5772381Z ##[command]/usr/bin/docker info -f "{{range .Plugins.Network}}{{println .}}{{end}}"
2024-07-19T09:24:25.6207711Z bridge
2024-07-19T09:24:25.6207822Z host
2024-07-19T09:24:25.6207924Z ipvlan
2024-07-19T09:24:25.6207964Z macvlan
2024-07-19T09:24:25.6208117Z null
2024-07-19T09:24:25.6208145Z overlay
2024-07-19T09:24:25.6230989Z ##[command]/usr/bin/docker network create --label dc4b27 vsts_network_6ef2b0cf21064e5bb3b329a9ec7e77c7
2024-07-19T09:24:25.7478058Z e2a0a1109677646b577e5e2d7f13d0b4b15861c21e418710bed640d2b2bf970e
2024-07-19T09:24:25.7627567Z ##[debug]Default Working Directory /__w/2/s/test-repo
2024-07-19T09:24:25.7627678Z ##[debug]Working Directory /__w/2/s
2024-07-19T09:24:25.7627728Z ##[debug]Mount Working Directory /opt/azagent/_work/2/s
2024-07-19T09:24:25.7636625Z ##[command]/usr/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" debian:bullseye
2024-07-19T09:24:25.7856637Z ##[command]/usr/bin/docker create --name 74f0a81dfd5d4cca841125cf336c07d2_debianbullseye_21184c --label dc4b27 --network vsts_network_6ef2b0cf21064e5bb3b329a9ec7e77c7  -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/opt/azagent/_work/2/s":"/__w/2/s" -v "/opt/azagent/_work/_temp":"/__w/_temp" -v "/opt/azagent/_work/_tasks":"/__w/_tasks" -v "/opt/azagent/_work/_tool":"/__t" -v "/opt/azagent/externals":"/__a/externals":ro -v "/opt/azagent/_work/.taskkey":"/__w/.taskkey" debian:bullseye "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
2024-07-19T09:24:25.8430675Z 3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9
2024-07-19T09:24:25.8468212Z DockerActionRetries variable value: False
2024-07-19T09:24:25.8471771Z ##[command]/usr/bin/docker start 3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9
2024-07-19T09:24:26.1242862Z 3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9
2024-07-19T09:24:26.1267037Z ##[command]/usr/bin/docker ps --all --filter id=3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
2024-07-19T09:24:26.1400945Z 3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 Up Less than a second
2024-07-19T09:24:26.1455841Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 sh -c "command -v bash"
2024-07-19T09:24:26.2147904Z ##[command]whoami 
2024-07-19T09:24:26.2164452Z build
2024-07-19T09:24:26.2164840Z ##[command]id -u build
2024-07-19T09:24:26.2177990Z 2028802
2024-07-19T09:24:26.2178556Z ##[command]id -g build
2024-07-19T09:24:26.2198522Z 2028801
2024-07-19T09:24:26.2199181Z ##[command]id -gn build
2024-07-19T09:24:26.2213430Z etm_devops_l
2024-07-19T09:24:26.2215041Z Try to create a user with UID '2028802' inside the container.
2024-07-19T09:24:26.2215595Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 bash -c "getent passwd 2028802 | cut -d: -f1 "
2024-07-19T09:24:26.3208190Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 bash -c "if [[ -e '/etc/alpine-release' ]]; then echo 'Alpine-based image detected.'; fi"
2024-07-19T09:24:26.3985418Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 groupadd -g 2028801 etm_devops_l_azpcontainer
2024-07-19T09:24:26.4990155Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 useradd -m -g etm_devops_l_azpcontainer -u 2028802 build_azpcontainer
2024-07-19T09:24:26.5795835Z Grant user 'build_azpcontainer' SUDO privilege and allow it run any command without authentication.
2024-07-19T09:24:26.5798090Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 groupadd azure_pipelines_sudo
2024-07-19T09:24:26.6925644Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 usermod -a -G azure_pipelines_sudo build_azpcontainer
2024-07-19T09:24:26.7822565Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 su -c "echo '%azure_pipelines_sudo ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers"
2024-07-19T09:24:26.8744356Z Allow user 'build_azpcontainer' run any docker command without SUDO.
2024-07-19T09:24:26.8744517Z ##[command]stat -c %g /var/run/docker.sock
2024-07-19T09:24:26.8763250Z 1001
2024-07-19T09:24:26.8763545Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 bash -c "cat /etc/group"
2024-07-19T09:24:26.9524939Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 groupadd -g 1001 azure_pipelines_docker
2024-07-19T09:24:27.0301262Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 usermod -a -G azure_pipelines_docker build_azpcontainer
2024-07-19T09:24:27.1341513Z ##[command]/usr/bin/docker exec  3b9824e64e396a2eb7339bd53b637c3f092485b56dfe866e52871640dacf20d9 bash -c "/__a/externals/node20_1/bin/node -v"
2024-07-19T09:24:27.2504923Z ##[section]Finishing: Initialize containers
KonstantinTyukalov commented 3 months ago

Thanks for the reporting! We are working on higher priority issues now, but we'll get back to this one soon

DenisNikulin5 commented 3 months ago

Hi @MaximilianGuttmann! This bug will be fixed in the next version of the agent. Thanks for the reporting!

gidad commented 2 months ago

Hi @DenisNikulin5 any update on this? The last agent version release notes v3.243.0 don't cite the current issue. Thank you

DenisNikulin5 commented 2 months ago

Hi @gidad! The release note for this is v3.243.0 Misc -> Remove AZP_AGENT_MOUNT_WORKSPACE knob (https://github.com/microsoft/azure-pipelines-agent/pull/4906)

We changed the behavior a bit to fix the bug :)