[Failing Test]: Many tests failing with invalid JWT signature

[kennknowles]

What happened?

• beam_CleanUpPrebuiltSDKImages
• beam_CleanUpGCPResources

ERROR: (gcloud.container.images.list) There was a problem refreshing your current auth tokens: ('invalid_grant: Invalid JWT Signature.', ***'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'***)
Please run:

> Task :beam-test-tools:removeStaleSDKContainerImages FAILED
  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account, run:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.

FAILURE: Build failed with an exception.

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• [ ] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [ ] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [ ] Component: Google Cloud Dataflow Runner

[kennknowles]

I think you have a fix up, yes? So I am assigning

[Abacn]

Yes I was trying to fix it - https://github.com/apache/beam/actions/workflows/beam_CleanUpGCPResources.yml?query=branch%3Atryfixpythontest

but not successful as some other affected workflows. Current plan to fix build-wheel first (which is more urgent one as it is used to build rc) then come back to these two

[Abacn]

The "Invalid JWT" error is affecting many performance tests that uses GitHub secret to set secret values also.

[Abacn]

CleanUpGCPResource currently permared due to two issues

• Some resources was created with some other credential, that the service account GitHub Action used doesn't have permission to delete them

• The JWT signature error observed in the test

For the second one, this happens since July 9th, and it is probablistic:

This suggests some roll out process is in play. Either GCP or GitHub is rolling out something that breaking the auth if conducted by "google-github-actions/auth@v1"

[Abacn]

Current status:

java_tests and python_tests failing, other tests fixed after switching to self-hosted runner's default cred

[Abacn]

downgrade to P2 as this has been mitigated. Due to that SA key expiration now enforced in the testing project, even if we generate new key and request update it, the key expires in 90 days and will break again.