Fix use of Zenodo in CI

fractal-analytics-platform / fractal-tasks-core

Main tasks for the Fractal analytics platform

https://fractal-analytics-platform.github.io/fractal-tasks-core/

BSD 3-Clause "New" or "Revised" License

12 stars 6 forks source link

Fix use of Zenodo in CI #568

Closed tcompa closed 10 months ago

tcompa commented 10 months ago

Checklist before merging

[x] I added an appropriate entry to CHANGELOG.md

tcompa commented 10 months ago

Current error is

E       urllib.error.HTTPError: HTTP Error 504: Gateway Time-out

Let's re-try later.

tcompa commented 10 months ago

My guess is that this PR is correct as is, and the CI errors are just due to slow network operations.

Zenodo just says

Oct 16 12:40 UTC: We have identified the cause of slow file uploads/downloads and will have it fixed in the coming 1-2 hours.

so let's try again tomorrow.

github-actions[bot] commented 10 months ago

Coverage report

The coverage rate went from 90.23% to 90.23% :arrow_right: The branch rate is 83%.

None of the new lines are part of the tested code. Therefore, there is no coverage data about them.

tcompa commented 10 months ago

Zenodo network issues are finally over.

Tests also seem to take significantly less time than before:

Worst pytest time in the latest CI of this PR: 5m15s (https://github.com/fractal-analytics-platform/fractal-tasks-core/actions/runs/6507611793?pr=568)
Worst pytest time in the latest green CI in main: 7m8s (https://github.com/fractal-analytics-platform/fractal-tasks-core/actions/runs/6479583007)

Nothing else changed in between, and then my guess is that the Zenodo download part was taking perhaps about a minute or two and now got faster. I cannot directly compare with the previous state, but I will add a log to know about download times as a future references (also in view of #565).

tcompa commented 10 months ago

For the record, here is an example of the time spent in downloading images/zarrs from Zenodo during the CI

My local network:

    Time spent in zenodo_images: 7.99 s
    Time spent in zenodo_zarr: 11.71 s

GitHub CI (EDIT: note that these times have important run-to-run fluctuations):

    Time spent in zenodo_images: 40.19 s
    Time spent in zenodo_zarr: 38.47 s

tcompa commented 10 months ago

Last comment: the Zenodo download times have important fluctuations (e.g. one of the two fixture may go from 15 to 81 seconds, in different run), which are likely the most important contribution to the GitHub-CI time fluctuations.