Download from a different workflow

[mastoj]

How do you download an artifact from another workflow?

A little bit of background. We have a build workflow that builds an artifact that we publish. Then we have a separate workflow that is triggered on deployment that needs to access this artifact. I tried to just use the same name for the artifact in download-artifact and upload-artifact hoping that it would download the files, that doesn't seem to work.

[mastoj]

It is a little bit unclear about the direction that you intended here. I definitely see some workarounds:

• Create a release and use the sha as tag. Don't like this approach since this will create a release of every build and mess up the releases.
• Use the package registry maybe? Can't do this right now since we don't have access

Maybe we are doing something completely wrong, and that is fine since it is quite new :). With that said, if we can't access the artifacts we have to revert to circle ci for the build and use github actions only for deploy.

[TingluoHuang]

@chrispat for feedback

[kimble]

This could also be relevant for build caching.

[cormacrelf]

And also for diffing artifacts between master and a pull request. There are hundreds of apps that do this for visual/webpage diffs, but none I can find that can diff a simple text file artifact.

[zlepper]

In addition to this: How to do this cross repositories?

Also we would like to limit the download to certain branches (Using somewhat the same logic as what TeamCity uses with their artifact dependencies), for example we can use the build output of one repository master, in another repository (creating a master build from several smaller projects). Preferably this would use the same branch if possible, or fallback to the default branch if possible.

[sjackman]

I'm very excited for this feature. I want to build and upload artifacts from a fork PR, and this part works well already. Fork PRs run without access to secrets, so in a second GitHub Action workflow that runs with access to secrets (outside of the fork PR), I want to download artifacts from GitHub and upload them to another service (BinTray). I had this arrangement working using CircleCI. This missing feature is blocking me from migrating from CircleCI to GitHub Actions, so I'm very keen to see it implemented!

[sjackman]

H, @ethomson! I work with @MikeMcQuaid on Homebrew. He told me that you may be a good person to give this issue some 👀. See also the related issue https://github.com/actions/upload-artifact/issues/21#issuecomment-562662657.

[mikkelbd]

Same use case as OP here. Any progress on this? 🙏 It would be really helpful to access an artifact uploaded from a build-workflow in a deploy-workflow. Especially since the cache action doesn't support deployment events: https://github.com/actions/cache/issues/63. Otherwise the cache could probably have been used with e.g. "artifact-name-$GITHUB_SHA" even though it's kind of a hack.

[chrispat]

@mikkelbd we now have an api for downloading artifactshttps://developer.github.com/v3/actions/artifacts/#download-an-artifact so you could likely accomplish your scenario now.

[eine]

@chrispat, I believe that @mikkelbd, as me and probably many others, is expecting this action to support the feature. In the end, if we are all forced to write custom scripts that interact with the API, all the "Actions" infrastructure is useless. This is specially unfortunate because GitHub's CI service as a product seems to be severely biased towards Actions.

[mikkelbd]

@chrispat Thank you for the prompt response :clap: Artifacts being available in the API is a good start. Combined with https://github.com/octokit/request-action or https://github.com/actions/github-script one could probably download it without writing custom code. However, it seems that you would have to know the internal artifact_id of the artifact, or you would have to list all artifacts and then filter by name? Again, then you would have to know the run_id that uploaded the artifact? I guess that's pretty hard in another workflow run...

As @eine stated, it would be a much better experience if the download-artifact action supported this out of the box instead of forcing n developers to interact with the GitHub API and also making the workflow-file bloated with low-level stuff 😅

[sjackman]

we now have an api for downloading artifactshttps://developer.github.com/v3/actions/artifacts/#download-an-artifact so you could likely accomplish your scenario now.

@chrispat Is there an API to get all the workflow runs for a given pull request or commit SHA-1? I found this API list-workflow-runs to list all the runs for a given repo and workflow, and that could then be filtered to find the SHA1 of the commit that you're interested in, but not terribly efficient.

https://developer.github.com/v3/actions/workflow_runs/#list-workflow-runs

[chrispat]

We don't currently have that parameter to the API but I will add it to our feedback list to consider for the future.

[sjackman]

pr="$(echo '${{github.event.head_commit.message}}' | sed 's/^.*#\([0-9]*\).*/\1/;q')"
git -C "$(brew --repo ${{github.repository}})" fetch origin "pull/$pr/head:pr"
sha1="$(git -C "$(brew --repo ${{github.repository}})" rev-parse pr)"
echo pr="$pr" sha1="$sha1"
run_id=$(curl -s -H 'Accept: application/vnd.github.antiope-preview+json' https://api.github.com/repos/${{github.repository}}/actions/workflows/build-bottles.yml/runs \
  | jq ".workflow_runs[] | select(.head_sha == \"$sha1\").id")
artifact_id="$(curl -s -H 'Accept: application/vnd.github.antiope-preview+json' https://api.github.com/repos/${{github.repository}}/actions/runs/$run_id/artifacts \
  | jq '.artifacts[0].id')"
echo run_id="$run_id" artifact_id="$artifact_id"
curl -L -o bottles.zip "https://${{secrets.GITHUB_PAT}}@api.github.com/repos/${{github.repository}}/actions/artifacts/$artifact_id/zip"

https://github.com/brewsci/homebrew-bio/blob/master/.github/workflows/upload-bottles.yml I've successfully used the new workflows and artifacts endpoints to download the artifact of given PR number in a push event! Yeah! Any comments or suggestions to make this easier would be welcome.

Are artifacts of a public repo public, and why is a GITHUB_PAT needed to download an artifact? GITHUB_TOKEN wasn't sufficient.

[jpfeuffer]

I found the same oddity. It seems like downloading artifacts requires "admin" permissions instead of "read" permissions (as documented).

It would really be great if this would be a standard feature of this official GitHub action. Is there any progress on this?

[josejulio]

I found the same oddity. It seems like downloading artifacts requires "admin" permissions instead of "read" permissions (as documented).

Is this reported somewhere? If not, where would be the ideal place to report it?

[davidobrien1985]

It would be great if we'd be able to access builds from other repos in our org through this action. Is that planned at all?

[josejulio]

It would be great if we'd be able to access builds from other repos in our org through this action. Is that planned at all?

This action doesn't do that yet, but you can do it with curl

see: https://github.com/RedHatInsights/policies-ui-frontend/blob/master/.github/scripts/download-latest-openapi.sh#L23

You need to pass the token in the workflow for it to work correctly: https://github.com/RedHatInsights/policies-ui-frontend/blob/master/.github/workflows/OpenAPIInSync.yml#L24

[mk-nickyang]

Here's an example of downloading artifact from a different workflow using github-script, hopefully it can help someone https://gist.github.com/mk-nickyang/c8094f6b32472eaebdd3c21b1ccb7e86

[fkirc]

I also believe that download-artifact is too restrictive, because sharing artifacts within a single workflow-run is not the only use case of Artifacts.

I am aware that there exists an artifacts-API that is separate from download-artifact. However, this API seems to lack name-addressable artifacts. Instead, this API requires some internal artifact_id. To get such an artifact_id, one has to either fetch a list of all artifacts in a repo, or fetch artifacts via the workflow-runs-API. But then again, it is painful to find id's for the workflow-runs-API.

To recap, all I want is to download an artifact-file with a specific name, where the artifact-file originates from a different workflow-run (either from the same or a different workflow file, this should not matter). The actions-API seems way too convoluted for this purpose; not to mention that this API uses ZIP-files and enforces actions to use secrets.GITHUB_TOKEN.

Personally, I won't bother to re-invent the wheel before something like https://github.com/actions/toolkit/pull/517 gets merged. In the meantime, I recommend to use services other than GitHub for artifact storage and deployment.

[josejulio]

I use this one to download artifacts from other repository: https://github.com/dawidd6/action-download-artifact Before that I used this script: https://github.com/RedHatInsights/policies-ui-frontend/blob/03870393c706db5d4f30d3708058bb3dd85241a2/.github/scripts/download-latest-openapi.sh

[fkirc]

Thanks for the hint, I will give it a try. For medium-sized projects, the scalability and reliability problems of the current solutions will hopefully not matter (think of large downloads or a huge number of Action-runs).

[baratgabor]

I'd also add my two cents in support of this feature. I hardly know anything about CI/CD, because I'm used to leads having that responsibility, but on my own project the very first thing I tried to do is to:

• build, test, and upload the tested build artifact in a build workflow (triggered by pull request on main), and then
• download the same artifact in a separate publish workflow, and publish it (triggered by push on main).

And I was surprised to find out that there is totally no built-in support for this, because I assumed this is a common thing to do.

[altendky]

@baratgabor, fwiw, I just put such jobs at the end of a single workflow and use needs: so they only run if everything else succeeds. Not the same, different tradeoffs, but you don't have to wait for this to get implemented.

[cedric05]

how do i down artifact for integration testing. it seems user has to be logged in for artifacts to be downloadable (for public repository)

artifacts:

• downloadable for logged in user releases:
• downloadable for anonymous user

Will this use case be covered, should i look for alternatives?

https://github.com/cedric05/dothttp-runner/blob/753fb1ab46b537d3c63027e0dc62e6090771a79e/version.json#L48-L56

[StephenCleary]

Being able to specify artifacts by run_id and name would make this common scenario much easier: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

Having to copy/paste the github-script everywhere isn't that great of a user experience, since most open-source repositories would benefit from the pull_request + workflow_run combo.

[oppknox]

We are at 606 days... does anyone work here?

[eine]

@oppknox see https://github.com/actions/upload-release-asset/issues/58.

[tiliv]

I'm leaving a +1 to be annoying. I'm shocked that downloading by workflow name has to be hand rolled by every single user when there's a perfectly good action right here.

[oppknox]

@tiliv Hey Autumn, I'm still annoyed that they have completely abandoned the community, but luckily I finally found a solution that ended up working for me. Praise be to dawidd6.

I highly recommend https://github.com/dawidd6/action-download-artifact. This paired well with a workflow dispatch job using RunId or if null use Latest/Success.

[ericoporto]

@eine do you have any inside information? Do you know if GitHub is killing the Actions feature?

[eine]

@ericoporto, I don't have any inside information. I am not nor ever was employed by GitHub, nor did I ever have access to any private repository or resource from the company. All my knowledge comes from pure observation of the public activities, announcements, talks.

I don't think GitHub is killing Actions under no circumstances. CI is strategically very important both for GitHub and Microsoft. Yet, they did never promise to provide useful and well-engineered actions/utilities. They announced the infrastructure and the pricing. That is exactly what we have. Most of the Actions were expected to be provided by the community, yet Actions were moved out of beta before some critical API features were made available. GitHub is not open source; it provides infrastructure that open source projects can use. Hence, GitHub had to provide some Actions, because users could not develop/improve them, even if they wanted, due to the lack of documentation. Since those critial pieces were documented, interest seems to be focused on https://github.com/cli/cli.

So, yes, some Actions in this namespace were born dead. Nonetheless, some other repos are very relevant and very used. It's the natural outcome of a mismatch between marketing and allocated resources. GitHub, as it's traditional with Microsoft, makes very easy to achieve the most basic tasks; however, the complex task are as complicated or more as using any other service. They target anyone in the widest sense, not experts only. I believe the request in this issue is almost irrelevant for a majority of GitHub Actions users. The experts that really need the feature found alternatives long time ago.

[ericoporto]

Thanks for the answer, I will keep it in mind. Guess I will delay moving from Azure to Actions for now.

[katlimruiz]

In my case, this is needed basically because Teams tier does not have Environments enabled.

I want to run build, wait for approval to enter staging, and then wait for approval to enter production. In Azure Pipelines I was able to do it in a single pipeline with three stages.

In GitHub Actions parlance, it would be one workflow, with three jobs (in serial), however, I am not able to stop execution of job 2 and job 3.

So I'm stuck.

I was very into moving to github actions, but it seems I won't do it and have to stay in Azure Pipelines.

[na-jakobs]

792 days since issue was opened :-/ we need this

[suecharo]

I didn't read all issues, but I used GitHub's command line tool (gh command) to download artifacts from other repositories' workflows in GitHub Actions as below:

- name: "Download artifact"
  run: |
    OTHER_REPO="owner/name"
    WF_NAME="wf_name"
    ARTIFACT_NAME="artifact_name"
    RUN_ID=`gh run --repo ${OTHER_REPO} list --workflow ${WF_NAME} --json databaseId --jq .[0].databaseId`
    gh run --repo ${OTHER_REPO} download ${RUN_ID} -n ${ARTIFACT_NAME}
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

[jszwedko]

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

[joahjoah]

Going by what @suecharo posted, I've added this implementation to my custom actions in order to share a JSON file, stored in an artifact, between workflows.

[Gby56]

I was just trying to figure out why I can't store an artifact in a workflow run, and download it in the next run, and that's why ? How is that not implemented ? this sounds like a crazy common use case to me, either to detect differences or reuse some build artifacts...

Would cache be more useful ? https://github.com/actions/cache I struggled to store a simple SARIF file and resorted to using upload artifact...

[joahjoah]

I was just trying to figure out why I can't store an artifact in a workflow run, and download it in the next run, and that's why ? How is that not implemented ? this sounds like a crazy common use case to me, either to detect differences or reuse some build artifacts...

Would cache be more useful ? https://github.com/actions/cache I struggled to store a simple SARIF file and resorted to using upload artifact...

That is what I use the above-mentioned gist for. Compare changes in test coverage from different workflow runs.

Using the Github API it is fairly straightforward to find previously-stored artifacts. You can use octokit.rest.actions.listArtifactsForRepo and octokit.rest.actions.downloadArtifact

@suecharo's snippet doesn't even require you to write a custom action for it.

[Gby56]

Yeah the problem with the gh run download cli is that it's not available on self hosted runners, the container image doesn't have it and I tried downloading the binary manually, I think I'm almost there but there is a remaining issue (probably caused bc the cli expects to be inside a repository, so I need to checkout...)

[demisx]

In the meantime, this custom action allows to download artifacts created in a different workflow.

[ashishjullia]

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

@jszwedko but how to get the value of run_id: ${{github.event.<workflow_run>.id }}?

Because in order to get artifacts we need to specify the run_id of the previous workflow1 (which uploads the artifacts) and need to download the uploaded artifacts (by id) say in worfklow2

I tried almost every possible way but can't make it run.

[MayCXC]

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

@jszwedko but how to get the value of run_id: ${{github.event.<workflow_run>.id }}?

Because in order to get artifacts we need to specify the run_id of the previous workflow1 (which uploads the artifacts) and need to download the uploaded artifacts (by id) say in worfklow2

I tried almost every possible way but can't make it run.

maybe https://github.com/actions/github-script/issues/262#issuecomment-1120167146 will help you?

[lpossamai]

Funny that we have to refer to 3rd parties for this. So many people would benefit from it.

[ashishjullia]

Just noting that Github has an example using github-script of downloading artifacts from another workflow in: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

@jszwedko but how to get the value of run_id: ${{github.event.<workflow_run>.id }}? Because in order to get artifacts we need to specify the run_id of the previous workflow1 (which uploads the artifacts) and need to download the uploaded artifacts (by id) say in worfklow2 I tried almost every possible way but can't make it run.

maybe actions/github-script#262 (comment) will help you?

Yup, this is what I'm using rn, works well.

[djbrown]

You can use context.payload.workflow_run.id inside an action script to get the triggering workflow run id. See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow.

[hwittenborn]

Is there any reason this hasn't been implemented yet? It seems like something that should be included, considering things like actions/checkout support specifying things like a repository just fine.

It looks like dawidd6/action-download-artifact will work just fine for getting this to all work, but having this work with a first-party action would definitely be ideal.

[AndrewDryga]

action-download-artifact doesn't download the last artifact the workflow, so you might end up using a custom script like we do: https://github.com/firezone/firezone/pull/2665

[philiplinell]

If you are not running self hosted runners (which may not have gh CLI), you can use a step such as the following to download an artifact across workflows.

  - name: Download artifact
    env:
      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    run: |
      gh run download --name my-artifact --dir ${{ github.workspace }}

gh is available on all GitHub-hosted runners https://docs.github.com/en/actions/using-workflows/using-github-cli-in-workflows.