Closed jauderho closed 3 years ago
To be clear, I have multiple actions that each build docker images for different apps failing, see https://github.com/jauderho/dockerfiles
I currently count 5 so it does not appear to an isolated issue.
same issue here, not on all images but some and its quite systematic
@markphelps Looks similar to #251
Just saw a suggestion on the orange site: https://news.ycombinator.com/item?id=28607735
I'm trying it out now and will report back.
As the ephemeral secrets.GITHUB_TOKEN
would be invalidated when the job is completed, I wonder if there is a limited token pool per repo? Any idea @chrispat?
We recently faced this; if you are using the docker/login action I'd give that a check as it turned out it was logging us out by default at the end of each job; resulting in some race conditions when running multiple runners on the same machine (sharing the same docker daemon).
As I can see you're not using a self-hosted runner so it's not your case.
Simple fix was to add
logout: false
to the action options.
Don't think it will change smth as docker logout
only erases token from the credential store.
Nope, it did not work. Still fails.
Also facing this issue, it seems to happen intermittently but more times than not.
I have verified the following steps work for me to resolve the issue, however, this is not ideal.
Maybe this can shed a light into what's happening @chrispat ?
Build that failed: https://github.com/k8s-at-home/container-images/runs/3656105847
Build that passed after deleting the image from GHCR: https://github.com/k8s-at-home/container-images/runs/3674834677
I have also tried toggling Inherit access from source repository (recommended)
and toggling the image visibility between Public and Private to no avail. This is definitely not an issue we cannot resolve on our end besides completely deleting the image and pushing it again.
I'm experiencing the same problem. I'm logging with docker-login-action
:
- name: Login to ghcr
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
And trying to push with this action:
- name: Build and push
uses: docker/build-push-action@v2
with:
builder: ${{ steps.buildx.outputs.name }}
context: ./docker
file: ./docker/Dockerfile
build-args: |
PHPSTAN_VERSION=dev-master#${{ github.sha }}
push: true
tags: ghcr.io/phpstan/phpstan:nightly
platforms: linux/amd64,linux/arm64
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
This is how the output tail looks like:
#27 [auth] phpstan/phpstan:pull,push token for ghcr.io
#27 DONE 0.0s
#28 [auth] phpstan/phpstan:pull,push token for ghcr.io
#28 DONE 0.0s
#29 [auth] phpstan/phpstan:pull,push token for ghcr.io
#29 DONE 0.0s
#26 exporting to image
#26 pushing layers 0.5s done
#26 ERROR: unexpected status: 403 Forbidden
------
> exporting to image:
------
error: failed to solve: unexpected status: 403 Forbidden
Error: buildx failed with: error: failed to solve: unexpected status: 403 Forbidden
After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.
It doesn't make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.
@onedr0p How are you getting to that dialog? My google-fu is failing me today and I am unable to find the location so that I can check that setting. Thanks.
You can find it in https://github.com/orgs/[XXX]/packages/container/[YYY]/settings
or by going to the package page then "package settings"
I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.
To reiterate, the steps are:
What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.
I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?
Update: All of my images have successfully built.
It works for me!
I'm also seeing this problem on push to a PR, but not on push for tags. So it might be somehow related to the metadata, which is different between PR push and tag?
In my case these were the tags for a build that didn't work:
Docker tags
ghcr.io/athensresearch/athens-server:pr-1684
but these tags worked:
Docker tags
ghcr.io/athensresearch/athens-server:mock-v0.1
ghcr.io/athensresearch/athens-server:latest
In my case the image did not exist at all to begin with, so I couldn't follow the steps in https://github.com/docker/build-push-action/issues/463#issuecomment-926229761.
@filipesilva
on push to a PR
Not related to that issue but check permissions of the GITHUB_TOKEN
in your workflow on PR. Afaik it will not work without a pull_request_target
event but be very careful with it.
Closing this issue as it seems to be solved.
Switching to a PAT worked for me. I used this action and redhat-actions/buildah-build
and redhat-actions/push-to-registry
and both resulted in a 403. Even with adding the repo to the package, granting explicit access, etc.
Once I returned to a PAT it worked flawlessly.
I think this 403 has to do with how the github packages registry resolves. Using any varient of ghcr.io
just in the tags or as the registry
property doesn't work. Using the registry: docker.pkg.github.com
how ever solves my issues.
I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.
To reiterate, the steps are:
1. Go to the packages page 2. Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings) 3. "Add a repository" - in my case, it is jauderho/dockerfiles and enable write access 4. Enable "Inherit access from source repository (recommended)"
What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.
I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?
Update: All of my images have successfully built.
I'm feeling dumb. Where is the packages page
?
@rupe120 it's on the user/org page. In your case:
4th tab on the top. https://github.com/rupe120?tab=packages
For me, that just says "get started with packages". There's no option to create a new one or anything.
I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.
To reiterate, the steps are:
- Go to the packages page
- Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings)
- "Add a repository" - in my case, it is jauderho/dockerfiles and enable write access
- Enable "Inherit access from source repository (recommended)"
What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.
I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?
Update: All of my images have successfully built.
For me none of the above worked, after wandering over the internet and github for 13 Hrs! I found out my solution [^1]:
Select the Read and write permissions
option for Workflow permissions
in the Actions' settings page of the repository (https://github.com/<username>/<repository>/settings/actions
).
[^1]: Note: I did this after ensuring steps 3 and 4 of @jauderho's comment were done.
For anyone who struggles to get to that settings page, two hints:
https://github.com/orgs/ORG NAME/packages/container/PACKAGE NAME/settings
For anyone stumbling into this error and have the same issue as @randomairborne, make sure that your personal token has write:packages
included in scope. That was the fix for me.
Apparently this error also triggers when you set an "invalid" tag.
As stated here, periods in tags are allowed.
The workflow step looked something like this:
- name: Build and push
uses: docker/build-push-action@v1
with:
push: true
load: false
file: path/to/Dockerfile
context: .
tags: ${{ env.DOCKER_TAGS }}
Where ${{ env.DOCKER_TAGS }} was something like this.is.my.tag
.
How is this tag invalid you ask? Well it seems like periods are not allowed in this case because this also triggered the above mentioned error. In my opinion this error is rather misleading!
After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.
It doesn't make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.
It works,Thank you.
What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.
I read somewhere in the GHA docs that if you publish a package via a workflow, but that workflow did not initially create the package, then it "would" or "could" fail -- don't remember which. I believe it had to do with how the package is linked to the repo. I tried to find these docs again but could not.
This issue should seriously be reopened. I've been struggling trying to fix this for at least three hours now, and I'm well versed in both Github actions and Docker.
I have no packages... so I can't access the basically hidden "package settings" link: for people reading this, you must click on any package that is listed if you have any, scroll two thirds down the page and you'll see on the right a "package settings" link.
My repo allows for all github actions to be executed.
And the GITHUB TOKEN has all permissions
So yup, here I am unable to figure out why this isn't working but really, this is a Github UX and Docker action problem.
Yeah i'm also having the same issue as @ChristopherRabotin how do we create a new package? When trying to push a new package that doesn't exist for the first time it just errors...
@mdsitton what's your yaml like? Despite the permissions above, I had to add the write permissions too : https://github.com/nyx-space/nyx/blob/master/.github/workflows/python.yml#L153
@mdsitton what's your yaml like? Despite the permissions above, I had to add the write permissions too : https://github.com/nyx-space/nyx/blob/master/.github/workflows/python.yml#L153
Probably not ideal to have write-all permission. I'm still seeing this very issue today.
I ended up deleting the package:
Instant satisfaction 🤷
thx @onedr0p (small world)
If you face this issue either someone already created a package with this name so it doesn't "inherit" from the repo or you may have inherit permissions off on the org/repo so you just have to do it yourself
Deleting the images created using another token worked for me.
Troubleshooting
Pushing to GHCR seems to randomly resulting in 403s in the last 24 hours. Rerunning the GH action seems to resolve things (partially).
Behaviour
See above.
Failure message :
buildx failed with: error: failed to solve: unexpected status: 403 Forbidden
Steps to reproduce this issue
Expected behaviour
Builds should complete with no issue.
Actual behaviour
See https://github.com/jauderho/dockerfiles/actions/runs/1255324536
Configuration