Buildkite CLI can not download Buildkite artifact

[gw0]

Is your feature request related to a problem? Please describe. The Buildkite CLI is failing to download artifacts. Official docs describe the command:

Downloading a specific file into a specific directory (note the trailing slash):
buildkite-agent artifact download build.zip tmp/

Describe the solution you'd like According to the docs to download the artifact myapp into go/myapp/myapp I would expect the following command to work (the destination is the directory with a trailing slash go/myapp/):

$ mkdir go/myapp
$ buildkite-agent artifact download myapp go/myapp/
2023-06-12 09:00:52 INFO   Searching for artifacts: "myapp"
2023-06-12 09:00:52 INFO   Found 1 artifacts. Starting to download to: /workdir/go/myapp
2023-06-12 09:00:52 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 1/5 Retrying in 5s
2023-06-12 09:00:57 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 2/5 Retrying in 5s
2023-06-12 09:01:02 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 3/5 Retrying in 5s
2023-06-12 09:01:08 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 4/5 Retrying in 5s
2023-06-12 09:01:13 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 5/5
2023-06-12 09:01:13 ERROR  Failed to download artifact: Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)
2023-06-12 09:01:13 FATAL  Failed to download artifacts: There were errors with downloading some of the artifacts

It is attempting to download the artifact to go/myapp instead of go/myapp/myapp. Of course the destination is a directory, this is why I explicitly indicated this with the trailing slash.

Describe alternatives you've considered I would expect that by not specifying the trailing slash, I can indicate the destination file go/myapp/myapp explicitly:

buildkite-agent artifact download myapp go/myapp/myapp
2023-06-12 08:51:29 FATAL  Failed to download artifacts: Could not find information about destination: /workdir/go/myapp/myapp stat /workdir/go/myapp/myapp: no such file or directory

Of course the destination file does not exist yet, because I have not downloaded it yet.

Additional context /

[gw0]

Official docs mention that recreating all files matching the uploaded file paths can be accomplished with:

Downloading all the files uploaded to log (including all subdirectories) into a local log directory (note that local directories will be created to match the uploaded file paths):
buildkite-agent artifact download "log/*" .

Unfortunately, moving to the destination directory and downloading to . also does not work:

+ cd go/myapp
+ buildkite-agent artifact download myapp .
2023-06-12 09:59:40 INFO   Searching for artifacts: "myapp"
2023-06-12 09:59:41 INFO   Found 1 artifacts. Starting to download to: /workdir/go/myapp
2023-06-12 09:59:41 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 1/5 Retrying in 5s
2023-06-12 09:59:46 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 2/5 Retrying in 5s
2023-06-12 09:59:51 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 3/5 Retrying in 5s
2023-06-12 09:59:56 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 4/5 Retrying in 5s
2023-06-12 10:00:01 WARN   Error trying to download https://mybucket/myapp (Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)) Attempt 5/5
2023-06-12 10:00:01 ERROR  Failed to download artifact: Failed to create file /workdir/go/myapp (*fs.PathError: open /workdir/go/myapp: is a directory)
2023-06-12 10:00:01 FATAL  Failed to download artifacts: There were errors with downloading some of the artifacts

[triarius]

Hi @gw0. Sorry that you are experiencing this. We are intending to overhaul how artifacts upload and download works. But in the meantime, can you contact support@buildkite.com and provide a build where this is happening so that we can debug it more thoroughly.

[gw0]

It turned out this are the consequences of some ill-designed logic in the Buildkite CLI. Docs describe it as: "If the last path component of matches the first path component of your , the last component of is dropped from the final path."

[triarius]

Gee, that's quite unexpected behaviour, but someone somewhere may be relying on it. It's quirks like this that have motivated our intention to replace the artifact upload/download subcommands with something that does not carry this baggage. However, that's going to take some time to develop.

If you're still blocked, let us know, and we can see what we can do in the interim.

[gw0]

The solution was to first check if checksum of already cached artifacts match, otherwise download to a temporary directory and then move to final destination.