Bitbucket legacy pipelines deprecated --> "authorization denied"

[AndyZe]

On Sep 17 I started experiencing this issue on my bitbucket pipelines:

time="2024-09-18T19:00:50.149245108Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "

I guess it's probably not a coincidence that bitbucket deprecated legacy pipelines on Sep 17.

There are some solutions described here

Basically it sounds like this is deprecated: DOCKER_BUILDKIT=0

Maybe the new way to do it is: export PATH=/usr/bin:$PATH

Here's my bitbucket-pipelines.yml with the fix I attempted. It didn't work.

image: docker:git

pipelines:
  default:
     - step:
         services:
           - docker
         script:
           # Attempted fix
           # See https://community.atlassian.com/t5/Bitbucket-questions/Docker-build-failing-for-buildkit-with-error-authorization/qaq-p/2377667#M94515
           - export PATH=/usr/bin:$PATH
           - apk add --update bash coreutils tar
           - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
           - .industrial_ci/bitbucket.sh ROS_DISTRO=humble

definitions:
  services:
    docker:
      memory: 2048

[mathias-luedtke]

Thanks for reporting!

time="2024-09-18T19:00:50.149245108Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "

Could you give me a little bit more context, please? This looks like the "docker pull" fails or gets blocked. You could set TRACE=true as well, which should print all the internal industrial_ci commands, which should help to narrow the scope.

Basically it sounds like this is deprecated: DOCKER_BUILDKIT=0

This has been deprecated (by docker) for a while now. And it does not matter, because industrial_ci does not build any docker images. It just runs them.

Maybe the new way to do it is: export PATH=/usr/bin:$PATH

I would not mess with $PATH unless there is a good reason to do that. To debug that more: Could you add an env call to your script and see what is the default for PATH. And does bitbucket set any DOCKER_* environment variables?

[mathias-luedtke]

According to https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/ running docker containers should be a supported use case. image: docker:git is not listed in any of the examples, though.

And it looks like bitbucket restricts the locations that can be mounted via volumes. Please try setting DOCKER_CREDENTIALS="". This should disable the extra-mounts, but as well the option to clone other repos.

If Docker does work anymore, there is as well the ISOLATION=shelloption: https://github.com/ros-industrial/industrial_ci/blob/master/.gitlab-ci.yml#L39

[AndyZe]

I added this stuff to bitbucket-pipelines.yml. Didn't seem to change anything, though-

image: docker:git
pipelines:
  default:
     - step:
         size: 2x # More memory
         services:
           - docker
         script:
           - export TRACE=true
           - env
           - export DOCKER_CREDENTIALS=""
           - git clone git@bitbucket.org:apptronik/elevate_robotics.git
           - apk add --update bash coreutils tar
           - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
           - .industrial_ci/bitbucket.sh ROS_DISTRO=humble
definitions:
  services:
    docker:
      memory: 4096

-->

Build tab ... Setting up ros-humble-ackermann-steering-controller (2.37.3-1jammy.20240911.165226) ... Setting up ros-humble-tricycle-steering-controller (2.37.3-1jammy.20240911.165229) ... Setting up ros-humble-bicycle-steering-controller (2.37.3-1jammy.20240911.165228) ... Setting up ros-humble-ros2-controllers (2.37.3-1jammy.20240911.165933) ... <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 'install_target_dependencies' returned with code '0' after 1 min 49 sec TRACE:util.sh:208 ici_hook after_install_target_dependencies

docker tab ... time="2024-09-19T15:17:33.492890622Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2024-09-19T15:17:33.492911474Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2024-09-19T15:17:33.493167953Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2024-09-19T15:17:33.650782870Z" level=info msg="ignoring event" container=5d9f648cf4e8cbd631df2e89534d58550cc785d2a7316d5f65c72b7b627f948e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" time="2024-09-19T15:17:33.650808975Z" level=info msg="shim disconnected" id=5d9f648cf4e8cbd631df2e89534d58550cc785d2a7316d5f65c72b7b627f948e namespace=moby time="2024-09-19T15:17:33.650862345Z" level=warning msg="cleaning up after shim disconnected" id=5d9f648cf4e8cbd631df2e89534d58550cc785d2a7316d5f65c72b7b627f948e namespace=moby time="2024-09-19T15:17:33.650873086Z" level=info msg="cleaning up dead shim" namespace=moby time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=false method=HEAD plugin=pipelines uri=/_ping time="2024-09-19T15:17:33.857952559Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: " time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/json time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/attach?stderr=1&stdout=1&stream=1" time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/wait?condition=removed" time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/start time="2024-09-19T15:17:33.905484049Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2024-09-19T15:17:33.905795247Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2024-09-19T15:17:33.908529910Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2024-09-19T15:17:33.908904756Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

[AndyZe]

Here's the env you asked for:

+ env
BITBUCKET_SSH_KEY_FILE=/opt/atlassian/pipelines/agent/ssh/id_rsa
BITBUCKET_REPO_UUID={413cc327-3443-45a1-bf0d-5a8f446b71c9}
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.34.224.1:443
CI=true
HOSTNAME=a70749b6-7bc4-4417-bf35-c6680ce44179-fk95q
BITBUCKET_STEP_TRIGGERER_UUID={d5ee4493-ec50-4736-919e-2b0c386ec051}
BITBUCKET_REPO_SLUG=aladdin_monitors
SHLVL=3
HOME=/root
BITBUCKET_GIT_SSH_ORIGIN=git@bitbucket.org:apptronik/aladdin_monitors.git
BITBUCKET_REPO_OWNER=apptronik
BITBUCKET_STEP_UUID={a70749b6-7bc4-4417-bf35-c6680ce44179}
BITBUCKET_BUILD_NUMBER=13
BITBUCKET_WORKSPACE=apptronik
BITBUCKET_CLONE_DIR=/opt/atlassian/pipelines/agent/build
BITBUCKET_GIT_HTTP_ORIGIN=http://bitbucket.org/apptronik/aladdin_monitors
BITBUCKET_REPO_IS_PRIVATE=true
PIPELINES_JWT_TOKEN=$PIPELINES_JWT_TOKEN
NPM_CONFIG_USER=65534
BITBUCKET_COMMIT=ebdf04f32a398291df1f1899931aaf35236962e8
DIND_COMMIT=65cfcc28ab37cb75e1560e4b4738719c07c6618e
BITBUCKET_PIPELINE_UUID={1316958a-3471-4095-b122-7802c08085d5}
BITBUCKET_DOCKER_HOST_INTERNAL=10.38.175.11
KUBERNETES_PORT_443_TCP_ADDR=10.34.224.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
BITBUCKET_REPO_OWNER_UUID={d5a60e11-1f8d-4fc5-adbf-f55f315a1124}
KUBERNETES_PORT_443_TCP_PROTO=tcp
BITBUCKET_STEP_RUN_NUMBER=1
BITBUCKET_BRANCH=main
DOCKER_VERSION=25.0.5
DOCKER_TLS_CERTDIR=/certs
BITBUCKET_REPO_FULL_NAME=apptronik/aladdin_monitors
DOCKER_HOST=tcp://localhost:2375
BITBUCKET_PROJECT_UUID={3462a40f-cc88-4642-86e5-cb294a61ebf7}
BITBUCKET_PROJECT_KEY=SCOR
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.34.224.1:443
DOCKER_BUILDX_VERSION=0.13.1
DOCKER_COMPOSE_VERSION=2.25.0
KUBERNETES_SERVICE_HOST=10.34.224.1
PWD=/opt/atlassian/pipelines/agent/build
TRACE=true

[AndyZe]

This may all be a red herring. I have other bitbucket pipelines that are still succeeding, same bitbucket-pipelines.yml. Very confused...

[mathias-luedtke]

According to your logs, industrial_ci is running and even managed to start the docker container and install things. What happens, if you remove "image: docker:git"?