crawler build fails because `azcopy sync` returns a 403

NotMyFault commented 8 months ago

Service(s)

trusted.ci.jenkins.io

Summary

The crawler job is failing again, this time azcopy sync yields a 403, causing the job to fail.

Reproduction steps

No response

dduportal commented 8 months ago

Audit:

https://github.com/jenkins-infra/crawler/blob/5afc48fa2cd3a7b225e8ef53b2815391f7c712cd/Jenkinsfile#L83
Most probably linked to a task of https://github.com/jenkins-infra/helpdesk/issues/3414

Checking...

dduportal commented 8 months ago

~As I don't know what @lemeurherve did in #3414 except the linked terraform PRs~ (see https://github.com/jenkins-infra/helpdesk/issues/3875#issuecomment-1868290946)

As we're in holidays for the next week, I've commented out the azcopy command https://github.com/jenkins-infra/crawler/commit/d0efde1404c2e45cc71737fc532025b8b325b061 and a build is currently running (I'll watch it).~

We'll work on this in January then

dduportal commented 8 months ago

And now

$ aws s3 sync ./updates/ s3://westeurope-updates-jenkins-io/updates/ --no-progress --no-follow-symlinks --size-only --exclude .svn --endpoint-url https://8d1838a43923148c5cee18ccc356a594.r2.cloudflarestorage.com/
09:50:50  fatal error: An error occurred (Unauthorized) when calling the ListObjectsV2 operation: Unauthorized

New hotfix incoming

dduportal commented 8 months ago

https://github.com/jenkins-infra/crawler/commit/e196bb7f25a92e2168f81f1130653d61f0c057b6 (watching build on trusted)

dduportal commented 8 months ago

Finished: SUCCESS we can finish 2023 in this state \o/

NotMyFault commented 8 months ago

Thanks for checking 🎄

dduportal commented 8 months ago

As I don't know what @lemeurherve did in #3414 except the linked terraform PRs, and because we're in holidays for the next week, I've commented out the azcopy command jenkins-infra/crawler@d0efde1 and a build is currently running (I'll watch it).

We'll work on this in January then

Nothing done in #3414 explains the HTTP/403. I've reopened #3818 and I'll check if I did not forget a firewall rule somewhere. That could also explain the aws s3 errors after the first hotfix.

dduportal commented 8 months ago

Resuming analysis on this topic.

https://github.com/jenkins-infra/azure/blob/240beb8f6f96fdf3bb114f66091120a971938481/updates.jenkins.io.tf#L15 shows that the bucket access is public. The 403 error cannot be caused by a storage account network restriction
Enabling again the azcopy on the crawler job to test: the mian branch job might have failure due to my tests. I'll also mention this in jenkins-infra (no status.jenkins.io as it is not a public thing)

dduportal commented 8 months ago

Retried and got the following detailed error message for the azcopy operation:

Time:2024-01-02T13:25:44.6252585Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Fri, 06 Oct 2023 00:00:00 GMT] - Expiry [Fri, 22 Dec 2023 00:00:00 GMT] - Current [Tue, 02 Jan 2024 13:25:44 GMT]</AuthenticationErrorDetail></Error>

dduportal commented 8 months ago

https://github.com/jenkins-infra/azure/blob/240beb8f6f96fdf3bb114f66091120a971938481/updates.jenkins.io.tf#L44 => gotcha.

dduportal commented 8 months ago

https://github.com/jenkins-infra/azure/pull/565 to rotate expiry

dduportal commented 8 months ago

Update:

Rotated the expiry of the SAS credential:
- https://github.com/jenkins-infra/azure/pull/565
- https://github.com/jenkins-infra/azure/commit/e86e885b2283a22c42906987b95020454fd11737

WiP:

Retrieve the new credential and apply it in trusted.ci
Check it work again for azcopy
Check error for the aws s3 sync and fix it if still present
Test access restriction for the SA

dduportal commented 8 months ago

Update:

Retrieve the new credential and apply it in trusted.ci ✅
Check it work again for azcopy ✅
Test access restriction for the SA => see https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-1874074475

WiP:

Check error for the aws s3 sync and fix it if still present
Add a calendar/updatecli process

dduportal commented 8 months ago

WiP: Check error for the aws s3 sync and fix it if still present

The error is fatal error: An error occurred (Unauthorized) when calling the ListObjectsV2 operation: Unauthorized when trying to S3-sync to the cloudflare bucket
There is no credential IaC configuration (AFAIR the cloudflare provider for terraform does not support - yet?- credentials generation) in https://github.com/jenkins-infra/cloudflare/blob/main/updates.jenkins.io.tf => next step, checking in the Cloudflare UI

dduportal commented 8 months ago

Gotcha: all API tokens have reached their TTL in Cloudflare. It also breaks the jenkins-infra/cloudflare project since last month (at least).

See https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-1874328160

dduportal commented 8 months ago

Update:

My hotfixes were reverted in crawler: rsync and azcopy are working fine, but aws s3 fails with UnAuthorized error
After https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-1877453959, a manual replay worked with success
It's failing again because I've added the IP restriction to the R2 token:
- If the requests come from trusted controller or permanent agent then it is fine
- However, the outbound IP for the ephemeral agents, since https://github.com/jenkins-infra/helpdesk/issues/3818, is changing. it means we have to add an outbound gateway in the new virtual network or link it to the existing gateway.

dduportal commented 8 months ago

Update:

Set up custom NAT gateways for the trusted.ci (and cert.ci) sponsored networks to keep the same outbound IP in https://github.com/jenkins-infra/azure/pull/567, https://github.com/jenkins-infra/azure-net/pull/187 and https://github.com/jenkins-infra/azure-net/pull/188
Ensured the R2 token can be used from the (new) outbound IP of trusted sponsored NAT gateway: https://github.com/jenkins-infra/terraform-states/commit/31b1e007037338d5aa5134feb106ed0da8cc45e6
Verified with a set manual build on crawler (2 build to check the outbound IP does not change and is the NAT gateway's, then 1 full build with success)
Automated SAS expiration date should be prepared (and notified) by updatecli with https://github.com/jenkins-infra/azure/pull/566

Todo:

Ensure crawler still works with SAS Ip restriction to be added

dduportal commented 8 months ago

Job still failing: https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-1879629619

dduportal commented 8 months ago

Last steps: https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-1879711494

Last build of crawler built without any problem

jenkins-infra / helpdesk