Open tcompa opened 10 months ago
Recent logging of the time spent in Zenodo downloads (see discussion in #568) made it clear that this operation takes a significant amount of time, in the GitHub CI (*). This could get to something like 2 minutes out of the 6 minutes spent in pytest
.
Let's keep this aspect in mind if/when we think about integrating other data sources in the CI.
I guess that we would get a significant speed up with a GitHub repo like fractal-test-data
, since connections would be all from GitHub (to be verified). This would be doable for our own data (the ones currently on Zenodo), but it's not obvious that we can do it for BIA or other sources.
(*) Note that this is not relevant for local tests, since Zenodo downloads only take place if the corresponding folders were removed from a standard location in tests/data
- which is typically not the case.
As part of #557, I added a small test that fetches data from https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD843, runs
import_ome_zarr
, and checks some aspect of the new image ROI table -- see code below.The test runs successfully (see https://github.com/fractal-analytics-platform/fractal-tasks-core/actions/runs/6494220845/job/17636746257?pr=557), but I will now mark it as "skip" (that is, it won't be run in the CI). This is because I don't know
I guess we should get in touch with the BIA team, and ask whether it is OK that we frequently fetch some small data (I think this zip file was approximately 5M-10M) in this way. If not, we can find other ways.