When using S3 for artifacts, `artifact-agent upload` behaves differently from `artifact_paths` in `pipeline.yaml`

[BadAsstronaut]

I discovered an interesting undocumented feature.

The following configuration works.

steps:
  - label: ':docker: / :node: - Build'
    key: build
    plugins:
      - ecr#v2.5.0:
          login: true
          region: 'us-east-1'
      - docker#v3.9.0:
          image: '<acct_id>.dkr.ecr.us-east-1.amazonaws.com/buildkite/docker/library/node:17'
          environment:
            - NPM_CONFIG__AUTH
            - NPM_CONFIG_REGISTRY
    commands:
      - 'cd /workdir'
      - 'npm install && npm run build'
      - 'tar --gzip -cf mock.tar.gz build'
      - buildkite-agent artifact upload mock.tar.gz

  - label: ':amazon-s3: - Publish'
    key: publish
    depends_on: build
    commands:
      - 'buildkite-agent artifact download mock.tar.gz .'
      - 'tar xmf mock.tar.gz'
      - './scripts/publish.sh build'
    plugins:
      - cultureamp/aws-assume-role#v0.2.0:
          role: arn:aws:iam:::role/cross-acct

This configuration does not work.

steps:
  - label: ':docker: / :node: - Build'
    key: build
    plugins:
      - ecr#v2.5.0:
          login: true
          region: 'us-east-1'
      - docker#v3.9.0:
          image: '<acct_id>.dkr.ecr.us-east-1.amazonaws.com/buildkite/docker/library/node:17'
          environment:
            - NPM_CONFIG__AUTH
            - NPM_CONFIG_REGISTRY
    commands:
      - 'cd /workdir'
      - 'npm install && npm run build'
      - 'tar --gzip -cf mock.tar.gz build'
     artifact_paths:
      - mock.tar.gz

  - label: ':amazon-s3: - Publish'
    key: publish
    depends_on: build
    commands:
      - 'buildkite-agent artifact download mock.tar.gz . --step build'
      - 'tar xmf mock.tar.gz'
      - './scripts/publish.sh build'
    plugins:
      - cultureamp/aws-assume-role#v0.2.0:
          role: arn:aws:iam:::role/cross-acct

Debugging the agent showed me that when using artifact_paths, the artifact is retrieved via the S3 client. If the step has a cross-account role without access to the artifact bucket, the step cannot access the artifact.

When creating an artifact use buildkite-agent artifact upload, the s3 uploader creates a presigned link which is accessible from the publish step using an assumed role.

This concerns me because the functionality is not documented and therefore subject to change.

[plaindocs]

Hi @BadAsstronaut thanks for opening this. We're updating the Artifact upload docs at the moment, and will look into it.

[moskyb]

hey @BadAsstronaut! are your artifacts stored in a private s3 bucket in your AWS account, or are you using buildkite's managed artifact storage?

also, would you be able to post some log output from a build that fails in this way? there are a couple of moving parts to artifacts which are tricky to diagnose without error messages

[BadAsstronaut]

@moskyb We are using the elastic stack on AWS and we have a private S3 artifacts bucket configured.

I don't want to share the logs because they include bucket names and such. I ended up using the --debug flag on the buildkite-agent artifact command and found that it retrieved the artifact via a presigned link. This is, imho, "desired" behavior. When the same was done with the artifact_paths config in the pipeline.yml file, the agent found the artifact but then failed with a Failed to download artifact: Could not s3:ListObjects in your AWS S3 bucket error.

This makes sense given that the step does not have a role enabling read access on the artifacts S3 bucket. But if the artifact is accessible via a presigned link, it doesn't need IAM access.

[moskyb]

@BadAsstronaut i'm gonna move this issue to the agent repo, as it's more of an agent issue than it is a docs issue. Even if it's a docs issue, it's an agent docs issue.

Further to that though, lemme look into this a bit deeper.