[question] Cache and Py* packages

tlienart commented 2 years ago

Hello,

I've been using the base GitHub cache to cache

~/.julia/artifacts
~/.julia/packages

with some success (similar to what you're doing here I think), however I encounter a lot of issues with PyCall/PyPlot. Did you ever try this?

Here are some of the combinations I tried:

setting ENV to "", forcing Conda.jl to be used, this works but it's slow, caching ~/.julia/conda works but it results in a massive cache (~900+mb)
setting ENV to "python" and using setup-python action + pip, optionally caching ~/cache/pip, this sometimes work and sometimes does not (...), many occurrences of either PyCall or PyPlot or both just not precompiling properly, no idea why; when it does work it's pretty fast
no ENV, no setup, no cache, this works but is slow

"slow" here means a 3-4 minute overhead, not the end of the world, just a bit annoying.

My hypothesis is that there's something clashing with how ./Julia* folders are cached and whether this affects PyPlot/PyCall's ability to find the correct PYTHON environment; whether I should try to force re-build these packages, not sure.

if you've never tried any of this or can't be bothered to figure it out (it's not really Julia after all), no problem at all, I understand. It's just been a fairly frustrating issue on my side 😓

Thanks in any case!

rikhuijzer commented 2 years ago

There are some instructions at actions/cache: https://github.com/actions/cache/blob/main/examples.md#python---pip and https://github.com/actions/cache/blob/main/examples.md#python---pipenv. Otherwise, I don't know but maybe someone else her has experience with it

tlienart commented 2 years ago

No I think the issue is that if you cache the ~/.Julia/packages/X for X in {PyCall, PyPlot, Conda} all hell breaks loose. This must be something to do with how these packages are built, looks like we can't cache that properly (at least I don't know how).

I'm currently doing

      - name: Cache 📖
        uses: actions/cache@v2
        with:
          path: |
                ${{ env.SITE_FOLDER }}/__cache
                ${{ env.SITE_FOLDER }}/__site
                ~/.julia/artifacts
                ~/.julia/packages
                !~/.julia/packages/PyPlot
                !~/.julia/packages/PyCall
                !~/.julia/packages/Conda

which works reliably but of course keeps the overhead :shrug: thanks for the input though!

SaschaMann commented 2 years ago

No I think the issue is that if you cache the ~/.Julia/packages/X for X in {PyCall, PyPlot, Conda} all hell breaks loose.

If we can verify that this is reliably the case, I think we should exclude those three packages from this action.

tlienart commented 2 years ago

Have a look at this super simple repo: https://github.com/tlienart/cache-pyplot there's only one yml file running dummy code with PyPlot.

initial run (no cache) no problem: https://github.com/tlienart/cache-pyplot/runs/5185496631?check_suite_focus=true
next run (loads cache) can't build PyCall/Conda/Pyplot: https://github.com/tlienart/cache-pyplot/actions/runs/1841678848
switching cache key (amounts to initial run) no problem: https://github.com/tlienart/cache-pyplot/runs/5185601840?check_suite_focus=true
next run (loads cache but not PyCall/PyPlot/Conda) also fails https://github.com/tlienart/cache-pyplot/runs/5185652801?check_suite_focus=true

So definitely the cache causes a problem but the exclusion of these packages is not sufficient to fix it...

Note: if anyone is reading this stuff and wondering what to do, this one https://github.com/tlienart/cache-pyplot/blob/eca991fa9c7fbe23cc12545d0a7f80bee0da6031/.github/workflows/exp.yml works and does the caching, it doesn't really bring a benefit in terms of time though (so might as well not use Cache here, but the cache might help for other stuff). It looks like it's required to re-build PyCall so that it gets a path setup properly.

rikhuijzer commented 2 years ago

Have a look at this super simple repo: https://github.com/tlienart/cache-pyplot there's only one yml file running dummy code with PyPlot.
* initial run (no cache) no problem: https://github.com/tlienart/cache-pyplot/runs/5185496631?check_suite_focus=true

* next run (loads cache) can't build PyCall/Conda/Pyplot: https://github.com/tlienart/cache-pyplot/actions/runs/1841678848

* switching cache key (amounts to initial run) no problem: https://github.com/tlienart/cache-pyplot/runs/5185601840?check_suite_focus=true

* next run (loads cache but not PyCall/PyPlot/Conda) also fails https://github.com/tlienart/cache-pyplot/runs/5185652801?check_suite_focus=true
So definitely the cache causes a problem but the exclusion of these packages is not sufficient to fix it...

Note: if anyone is reading this stuff and wondering what to do, this one https://github.com/tlienart/cache-pyplot/blob/eca991fa9c7fbe23cc12545d0a7f80bee0da6031/.github/workflows/exp.yml works and does the caching, it doesn't really bring a benefit in terms of time though (so might as well not use Cache here, but the cache might help for other stuff). It looks like it's required to re-build PyCall so that it gets a path setup properly.

Then it seems that Conda.jl creates ~/conda somewhere during the build step and then assumes that it will exist from that point on. Would be best if that would be solved at the Conda.jl side

tlienart commented 2 years ago

it's tricky, caching (or not) ~/.Julia/conda doesn't help. The only thing that helps is to explicitly rebuild PyCall. I'll try raising the issue with PyCall and see..

SaschaMann commented 1 year ago

Given that this comes up frequently (#42, #47), I wonder if we should exclude those packages from being cached (either by default or optionally), effectively implementing the workaround from above. They're popular enough that special-casing them might be worth it.

Any thoughts on this?

giordano commented 1 year ago

Might be worth it. Do we know why they are problematic to cache?

SaschaMann commented 1 year ago

From the comments above:

It looks like it's required to re-build PyCall so that it gets a path setup properly.

Then it seems that Conda.jl creates ~/conda somewhere during the build step and then assumes that it will exist from that point on. Would be best if that would be solved at the Conda.jl side

Why that is the case, no idea.

axsk commented 8 months ago

I think I am running into similar problems with using ENV["PYTHON"]="" Every second build (i.e. the one when the cache is loaded) I get an error

ERROR: LoadError: InitError: could not load library "/home/runner/.julia/conda/3/x86_64/lib/libpython3.10.so.1.0"
/home/runner/.julia/conda/3/x86_64/lib/libpython3.10.so.1.0: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen
    @ Base.Libc.Libdl ./libdl.jl:116 [inlined]
  [3] __init__()
    @ PyCall ~/.julia/packages/PyCall/1gn3u/src/pyinit.jl:149

Then the cache is removed, and the next run works fine again (but takes 30mins without caching..)

IanButterworth commented 8 months ago

Please add your comment to this issue https://github.com/JuliaPy/Conda.jl/issues/251

julia-actions / cache

[question] Cache and Py* packages #10