search

JuliaPy / Conda.jl

Conda managing Julia binary dependencies
Other
174 stars 57 forks source link

Use scratchspaces rather than .julia/conda #251

Open IanButterworth opened 10 months ago

IanButterworth commented 10 months ago

scratchpaces have become the standard place for things like that which are currently stored in .julia/conda. Notably CondaPkg.jl uses scratchspaces.

Also, using a custom dir means special handling would be needed for tools like julia-actions/cache

Krastanov commented 7 months ago

Moving to scratchspaces will probably also fix CI issues where for some reason DEPOT/conda/3/x86_64 does not exist. It happens very often in CI and it seems the standard fix is to add mkdir -p "DEPOT/conda/3/x86_64" to CI scripts.

These issues are not reproducible locally for me, only in CI.

E.g. these two lines are necessary in my CI scripts:

Without them, this happens at runtime:

image

I learnt of this workaround from slack where other people were complaining about similar issues. Copied here for archival:

image

Here is the link to the Zygote issue:

IanButterworth commented 7 months ago

Yes, either: 1) This issue needs to be fixed 2) https://github.com/julia-actions/cache/issues/72 should be implemented 3) Conda.jl should somehow detect at runtime, after precompilation, if it needs to reinstall things

But note that 2) would require users to opt-in.

Doing 3) would probably be a good thing either way.

Until then users of Conda.jl are going to have failing because CI caching is becoming more widespread, so it would be wise to consider this an urgent issue

Krastanov commented 7 months ago

Just a minor addendum: some of these CI issues I mentioned above are independent of julia-actions/cache (e.g. the ones for buildkite)

axsk commented 7 months ago

I was directed here with the same issue, namely ERROR: LoadError: InitError: could not load library "/home/runner/.julia/conda/3/x86_64/lib/libpython3.10.so.1.0" /home/runner/.julia/conda/3/x86_64/lib/libpython3.10.so.1.0: cannot open shared object file: No such file or directory on every second CI run (when the cache is loaded). Every other run the cache was deleted and it works fine. Will look into the workaround, thank you @Krastanov