Artifact download status flag when searching/downloading

[nmathur94]

Is your feature request related to a problem? Please describe.

Using Elastic stack v5.7.2, we have some steps that we dynamically upload based on a delay. This allows us to build and upload an artifact and at the same time, warm up agents that consume this artifact.

This works great however, updating to v5.9.0 and the associated agent baked into the AMI, we now experience some issues with downloading that artifact. At the moment, we have a polling loop running that will check if the artifact exists or not. If it doesn't, retry again in 60 seconds. Once the artifact exists, we then download and proceed.

After reaching out to support, found out that there is now a parameter passed in the api call that explicitly checks for finished artifacts only. #1584 The change does make sense, however if a search comes back with results, download should be able to get it. Before this, if the artifact was 96% uploaded when executing the download command, it would retry after a couple of seconds automatically and continue.

Describe the solution you'd like If search gives us results that an artifact exists, download should be able to get it without any issues. If you want to search for non-complete artifacts and also try downloading them, allow some control through a parameter, but defaulting to completed artifacts as per the pr mentioned.

If partial parameter is given and for example, the artifact is 96% uploaded when the download command is executed, retry again in a few seconds automatically up to 5 times.

Describe alternatives you've considered At the moment I have added a sleep of 25 seconds between artifact search and download commands. It will detect it exists, and it will try the download after 25 seconds which is enough time for it to finish uploading.

Additional context Just before we execute asset build steps, we run the following:

echo '--- :cypress: :jest: Upload steps'
(sleep 120 ; buildkite-agent pipeline upload .buildkite/build-assets-pipeline.yaml) &

This will push additional steps to the pipeline after 120 seconds. The agents will warm up and be ready to consume the artifact almost instantly. For example, execute yarn install && yarn prod right after the pipline upload command, and steps will get pushed to the build while yarn is doing its thing.

How the artifact is downloaded

// Search once, check output matches filename
until buildkite-agent artifact search $artifactName --build ${BUILDKITE_BUILD_ID} --format "%p" | grep $artifactName
do
  sleep 60 // No match found so try in 60 seconds
done
sleep 25 // Temporary sleep as it finds artifact but is 99% uploaded
// Previous agent version would retry, now this step exits with FATAL error, artifact doesnt exist without the 25 second sleep
buildkite-agent artifact download $artifactName . --build ${BUILDKITE_BUILD_ID}

Before Agent update: would retrieve immediately or do a retry as shown below

After Agent update (without 25 second sleep in-between):

[jradtilbrook]

I think to resolve this, buildkite-agent artifact search needs to accept an extra flag on whether to search for only finished uploads the same way buildkite-agent artifact download does. That way, users can poll with the search and download afterwards which is currently not working.

Alternatively, another command could be introduced; buildkite-agent artifact exists which does the same thing but doesn't need to add a flag to the search command`.

[pda]

IMO the default of buildkite-agent artifact search should change to finished-only, and if anything there should be a flag to opt in to including unfinished artifacts.

[sj26]

As a workaround could you poll for meta-data which is set once the artifact is uploaded?