Closed danking closed 2 months ago
Same issue with pyproject-metadata. I'm guessing this is due to attestations: true
(since scikit-build-core just released with no issues), maybe due to the attestation change that was merged at https://github.com/pypi/warehouse/pull/16757 but is still pending here at https://github.com/pypa/gh-action-pypi-publish/pull/262 ? Not a very helpful error if that's the case.
cc @woodruffw @di ^
@danking that looks like a PyPI issue so we need to wait for the Warehouse maintainers to take a look. Meanwhile, if Henry is right, disabling the attestations might be a workaround for you.
I missed #262 in my notification. Thanks for pointing it out! I'll go ahead and merge it just in case that's it..
@danking could you try the commit from William's fork?
@danking restarting your workflow should pick up the new release. Let me know if #262 fixed it…
I don't see a release? https://github.com/pypa/gh-action-pypi-publish/releases
I work with @danking . Just rerun it https://github.com/spiraldb/vortex/actions/runs/10962752610/job/30454969277. I see pypi-attestations~=0.0.12 but the response is an error (though not 502)
INFO Response from https://upload.pypi.org/legacy/:
400 Could not verify the uploaded artifact using the included
attestation: Verification failed: 0 of 2 policies succeeded
INFO <html>
<head>
<title>400 Could not verify the uploaded artifact using the included
attestation: Verification failed: 0 of 2 policies succeeded</title>
</head>
<body>
<h1>400 Could not verify the uploaded artifact using the included
attestation: Verification failed: 0 of 2 policies succeeded</h1>
The server could not comply with the request since it is either
malformed or otherwise incorrect.<br/><br/>
Could not verify the uploaded artifact using the included attestation:
Verification failed: 0 of 2 policies succeeded
</body>
</html>
ERROR HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Could not verify the uploaded artifact using the included attestation:
Verification failed: 0 of 2 policies succeeded
Ahh, it was pushed to the release branch. Better error at least!
uses: ./.....
@robert3005 @danking reusable workflows are still unsupported: https://github.com/pypa/gh-action-pypi-publish/discussions/255#discussioncomment-10630881. That's what you're hitting. Follow the issue mentioned there to get notified when it's implemented.
I've already uploaded the pyproject-metadata wheel manually, so I can't test quickly. I can see if there are any I can quickly check.
I don't see a release?
@henryiii I currently push a signed tag (and branches updates) from my laptop, and only then I fill out the GH release semi-manually. That's why you're seeing it appearing a bit later.
I can see if there are any I can quickly check.
Thanks! I suppose if there's a bigger problem, we'll get more reports.
I also checked that https://github.com/pypi/warehouse/pull/16757 was merged yesterday and since this is the first report we're seeing, my best guess is that this report is not related to that PR.
At least some closure. I can refactor and inline our workflows
Hmm. @webknjaz, I did anticipate that reusable workflows would complicate the situation and actually granted both workflows (the caller and callee) trusted status. Regardless, I'll inline the workflow.
@danking reusable workflows sometimes work for some people in some narrow cases, but that's mostly by accident. Warehouse still needs to implement this properly. IIRC, you need to have both workflows in the same repository and use secrets: inherit
. But that's just off the top of my head, I'm not using this approach and waiting until it's implemented officially. I just know that I saw it work somewhere, somehow.
Thanks for verifying!
Why does the changelog say "nah, that wasn't it?" It did fix the issue, which was affecting every user with attestations: true
; the secondary issue about reusable workflows is separate.
I was under the impression that it was just a coincidence.. Perhaps I was wrong. I thought so because there were no reports for like a day after the Warehouse change.
Yep, that was indeed the fix! I think the reason no other reports occurred (thankfully) is because most people haven't flipped the attestations:
switch 😄
I'm not sure I follow why the difference in pypi-attestations
version caused a 500-level error here, @woodruffw do you know? I also didn't see these errors surface in our error reporting either, I think.
Hmm, you're right -- it should have caused solely a 400-level error, since the change in pypi-attestations
was purely a fix to the Pydantic models (which, when validation fails, get surfaced via 4xx).
502 suggests a transient Fastly or other edge routing failure, possibly?
Yes, that is generally what I would expect to be the cause of a 502.
FWIW, from 2024-09-20 14:33 UTC until around 2024-09-20 21:51 UTC, we ran the GitHub Action to submit to PyPI ten times. Each time it retries five times, each time receiving 502 Bad Gateway. I'm not sure exactly from which attempt the first log came.
We likewise assumed it was transient, which is why we retried for a few hours before reporting. Seems like maybe something between us and the PyPI servers misinterpreted the attestation failure as a 502?
Weird. There was a Fastly incident right around then, but it looks like it was in an unrelated service + was already being resolved by the time you began seeing the 502s here: https://www.fastlystatus.com/incident/376938
I pushed several releases during that time, and the ones with attestations all failed. The ones without all succeeded.
Most packages aren't using attestations: true
, as it's new and states its a beta feature.
Easy way to test: just pin the old one and try it. :)
@woodruffw @di is it possible that Warehouse was responding with a 4xx and Fastly was turning that into a 502 by mistake as Dan suggested?
I don't see any way that that would happen, no.
Pinning 1.10.1 absolutely does still produce a 502:
https://github.com/henryiii/validate-pyproject-schema-store/actions/runs/11224641446/job/31201797202
Uploading distributions to https://upload.pypi.org/legacy/
Uploading validate_pyproject_schema_store-2024.10.7-py3-none-any.whl
WARNING Received "502: Bad Gateway"
Package upload appears to have failed. Retry 1 of 5.
Uploading validate_pyproject_schema_store-2024.10.7-py3-none-any.whl
WARNING Received "502: Bad Gateway"
Package upload appears to have failed. Retry 2 of 5.
Uploading validate_pyproject_schema_store-2024.10.7-py3-none-any.whl
WARNING Received "502: Bad Gateway"
Package upload appears to have failed. Retry 3 of 5.
Uploading validate_pyproject_schema_store-2024.10.7-py3-none-any.whl
WARNING Received "502: Bad Gateway"
Package upload appears to have failed. Retry 4 of 5.
Uploading validate_pyproject_schema_store-2024.10.7-py3-none-any.whl
WARNING Received "502: Bad Gateway"
Package upload appears to have failed. Retry 5 of 5.
WARNING Error during upload. Retry with the --verbose option for more details.
ERROR HTTPError: 502 Bad Gateway from https://upload.pypi.org/legacy/
Bad Gateway
And unpinning fixes it:
https://github.com/henryiii/validate-pyproject-schema-store/actions/runs/11224728491/job/31202070560
Uploading distributions to https://upload.pypi.org/legacy/
Uploading validate_pyproject_schema_store-2024.10.7-py3-none-any.whl
Uploading validate_pyproject_schema_store-2024.10.7.tar.gz
View at:
https://pypi.org/project/validate-pyproject-schema-store/2024.10.7/
So I think the changelog should be re-edited to say this indeed fixed it.
@henryiii thank you! fixed.
That's surprising, @woodruffw we should probably figure out why that's happening...
That's surprising, @woodruffw we should probably figure out why that's happening...
I suspect it's the base64 change that caused us to do that point release in the first place, although I'm confused as to how it got surfaced as a 502. I'll look into it!
I'm confused as to how it got surfaced as a 502. I'll look into it!
I looked into it a bit, and I don't have a great working theory for why this is a 502 instead of a 40x. The only thing I can really think of is that the upload endpoint is on the forklift
domain (i.e. upload.pypi.org
) instead of pypi.org
, so maybe there's some default exception processing that's being suppressed. I'll keep looking at it, though, since that doesn't seem very likely (I can't find anything in Warehouse or the infra routing that would suggest that theory).
I'm not sure if I should report here or on some PyPI specific place, but every time this action is triggered, we get a 502. I am able to publish (the exact same wheels) using twine from my laptop.
This is the whole GitHub Action output with debug logs. I can get you the wheels if that's useful. Are you able to see the GitHub Action page?