issue requests in gdrive/fsdi upload

[davidoesch]

Describe the bug On production in GithubAction we have the following errors since 7-11-2024, it seems to be an issue when accesing the STAC API ( not as initially thought related to gdrive) it hangs in https://github.com/swisstopo/topo-satromo/blob/2f06e5e45f4997bd2b817986b775e1ae7c444e27/main_functions/main_publish_stac_fsdi.py#L537

1. Thu, 07 Nov 2024 03:34:18 GMT

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) https://github.com/swisstopo/topo-satromo/actions/runs/11715976429/job/32633284773#step:9:372

2. Fri, 08 Nov 2024 04:38:35 GMT

File "/home/runner/work/topo-satromo/topo-satromo/main_functions/main_publish_stac_fsdi.py", line 537, in upload_asset Fri, 08 Nov 2024 04:38:35 GMT response.json()['urls'][0]['url'], data=data, headers=***'Content-MD5': md5***) Fri, 08 Nov 2024 04:38:35 GMT KeyError: 'urls' https://github.com/swisstopo/topo-satromo/actions/runs/11735443612/job/32693135080#step:9:270

3.Sat, 09 Nov 2024 03:21:00 GMT File "/home/runner/work/topo-satromo/topo-satromo/main_functions/main_publish_stac_fsdi.py", line 537, in upload_asset Sat, 09 Nov 2024 03:21:00 GMT response.json()['urls'][0]['url'], data=data, headers=***'Content-MD5': md5***) Sat, 09 Nov 2024 03:21:00 GMT KeyError: 'urls' https://github.com/swisstopo/topo-satromo/actions/runs/11752851798/job/32744992770#step:9:37

observation

• Cannot be reproduced on DEV
• only occurs with VHI files so far, so at the end of the upload
• Data is exported on time around 2330 while publisher fails at 0400. Data is avilable and is processed

Possible cause

• [X] requests ist outdated and so there is an issue with , updated https://github.com/swisstopo/topo-satromo/commit/fda021a0790cfab4a0cd24f5b7bce55104f1c979 , no effect
• [ ] latency in github conenction with bgdi infra?
• [ ] STAC API overloaded?

To Reproduce Steps to reproduce the behavior:

1. run VHI operationally on prod

Expected behavior Runs as on dev

[davidoesch]

When copying locally data to PROD gdrive

vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

maybe this is the issue, accorindg to https://forum.rclone.org/t/failed-to-move-googleapi-error-403-user-rate-limit-exceeded-userratelimitexceeded/44068/2 there might be a daily limit on gdrive