Consider using Artifacts for data

JuliaOcean / Climatology.jl

Climatology, including of the Ocean state

https://JuliaOcean.github.io/Climatology.jl

MIT License

11 stars 2 forks source link

Consider using Artifacts for data #6

Closed vchuravy closed 3 years ago

vchuravy commented 3 years ago

PackageEvaluator.jl runs nightly tests over the package ecosystem and looking at our traffic numbers I noticed that IndividualDisplacements downloads 1.747GB of data during it's tests. Using Artifacts.jl would allow us to cache that.

gaelforget commented 3 years ago

Hopefully what I did in https://github.com/gaelforget/OceanStateEstimation.jl/pull/8 will be helpful.

Looking into Artifacts for this had been on my to do list. Thanks for bringing it up.

Not sure I totally understand how best to handle my use case though (many files, downloaded from DOI'ed archive, not in tar ball, downloaded by parts). I feel that what I did is a bit of a hack but it seems to work ok. Will try using this in IndividualDisplacements after new release is completed

vchuravy commented 3 years ago

Thanks!

gaelforget commented 3 years ago

I think I finished moving data dependencies to the Artifacts system as of v0.2.1 of IndividualDisplacements.jl (released 5 days ago).

Some of the downloads associated with the artifacts still happen manually but at least they go to ~/.Julia/artifacts

Am wondering how much of an impact this had on the PackageEvaluator.jl runs -- could you please check @vchuravy when you get a chance?

vchuravy commented 3 years ago

Looks better: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2020-11/10/OceanStateEstimation.1.6.0-DEV-9c498bff43.log

Network usage:
- eth0: 126.608 MiB received, 708.351 KiB sent