The retraction script has been failing regularly for about 11 days now. There seem to be two separate issues.
One is a known issue of a search node timing out (because the amount of returned items is less than the expected items - this seems to be happening only infrequently in the past for reasons I do not fully understand.
But there are also a bunch of failures like this which show a trace similar to here
```shell
Backing up catalog
[260](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:261)
Traceback (most recent call last):
[261](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:262)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/retry.py", line 115, in retry_request
[262](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:263)
return await func(*args, **kwargs)
[263](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:264)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 374, in _request
[264](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:265)
headers=self._get_headers(headers),
[265](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:266)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 353, in _get_headers
[266](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:267)
self.credentials.apply(out)
[267](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:268)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 182, in apply
[268](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:269)
self.maybe_refresh()
[269](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:270)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 177, in maybe_refresh
[270](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:271)
self.credentials.refresh(req)
[271](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:272)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/service_account.py", line 410, in refresh
[272](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:273)
access_token, expiry, _ = _client.jwt_grant(
[273](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:274)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 289, in jwt_grant
[274](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:275)
response_data = _token_endpoint_request(
[275](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:276)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 260, in _token_endpoint_request
[276](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:277)
_handle_error_response(response_data, retryable_error)
[277](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:278)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
[278](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:279)
raise exceptions.RefreshError(
[279](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:280)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', ***'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'***)
[280](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:281)
Traceback (most recent call last):
[281](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:282)
File "create_filtered_catalog.py", line 75, in
[282](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:283)
gcs.put_file(local_filename, f'cmip6/***backup_filename***')
[283](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:284)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/fsspec/asyn.py", line 111, in wrapper
[284](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:285)
return sync(self.loop, func, *args, **kwargs)
[285](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:286)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/fsspec/asyn.py", line 96, in sync
[286](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:287)
raise return_result
[287](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:288)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in _runner
[288](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:289)
result[0] = await coro
[289](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:290)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 1061, in _put_file
[290](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:291)
location = await initiate_upload(
[291](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:292)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 1559, in initiate_upload
[292](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:293)
headers, _ = await fs._call(
[293](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:294)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 392, in _call
[294](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:295)
status, headers, info, contents = await self._request(
[295](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:296)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/decorator.py", line 221, in fun
[296](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:297)
return await caller(func, *(extras + args), **kw)
[297](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:298)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/retry.py", line 147, in retry_request
[298](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:299)
raise e
[299](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:300)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/retry.py", line 115, in retry_request
[300](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:301)
return await func(*args, **kwargs)
[301](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:302)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 374, in _request
[302](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:303)
headers=self._get_headers(headers),
[303](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:304)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 353, in _get_headers
[304](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:305)
self.credentials.apply(out)
[305](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:306)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 182, in apply
[306](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:307)
self.maybe_refresh()
[307](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:308)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 177, in maybe_refresh
[308](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:309)
self.credentials.refresh(req)
[309](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:310)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/service_account.py", line 410, in refresh
[310](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:311)
access_token, expiry, _ = _client.jwt_grant(
[311](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:312)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 289, in jwt_grant
[312](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:313)
response_data = _token_endpoint_request(
[313](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:314)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 260, in _token_endpoint_request
[314](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:315)
_handle_error_response(response_data, retryable_error)
[315](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:316)
File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
[316](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:317)
raise exceptions.RefreshError(
[317](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:318)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', ***'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'***)
```
The retraction script has been failing regularly for about 11 days now. There seem to be two separate issues.
I think the latter might be caused by our quite outdated version of google-github-actions/auth.
I will try to activate dependabot for this repo and see if that gets rid of these failures at least.