search

JuliaPy / Conda.jl

Conda managing Julia binary dependencies
Other
171 stars 57 forks source link

disable auto_update_conda?: conflict conda versions when installing RCall.jl and PyCall.jl #242

Closed szcf-weiya closed 10 months ago

szcf-weiya commented 1 year ago

I encountered the error in the AutoMege step when registering my package that depends on RCall.jl. Here is the pull request on the General repo and the action log info.

https://github.com/JuliaRegistries/General/pull/86838 https://github.com/JuliaRegistries/General/actions/runs/5464785321/jobs/9947294633?pr=86838

    Building Conda ─→ `/tmp/jl_aWu15K/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/915ebe6f0e7302693bdd8eac985797dba1d25662/build.log`
    Building PyCall → `/tmp/jl_aWu15K/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/43d304ac6f0354755f1d60730ece8c499980f7ba/build.log`
    Building RCall ─→ `/tmp/jl_aWu15K/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/d441bdeea943f8e8f293e0e3a78fe2d7c3aa24e6/build.log`
ERROR: Error building `RCall`: 
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: ...working... unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.

ResolvePackageNotFound: 
  - conda==23.1.0

[ Info: Installing R via Conda.  To use a different R installation, set the "R_HOME" environment variable and re-run Pkg.build("RCall").
[ Info: Running `conda config --add channels r --file /tmp/jl_aWu15K/conda/3/x86_64/condarc-julia.yml --force` in root environment
[ Info: Running `conda install -y 'r-base>=3.4.0,<4'` in root environment
ERROR: LoadError: failed process: Process(setenv(`/tmp/jl_aWu15K/conda/3/x86_64/bin/conda install -y 'r-base>=3.4.0,<4'`,["PATH=/opt/hostedtoolcache/julia/1.9.1/x64/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin", "CONDA_PREFIX=/tmp/jl_aWu15K/conda/3/x86_64", "***", "DISPLAY=:99", "JULIA_LOAD_PATH=@:/tmp/jl_8YYRpk", "OPENBLAS_DEFAULT_NUM_THREADS=1", "JULIA_REGISTRYCI_AUTOMERGE=true", "XAUTHORITY=/tmp/xvfb-run.ZuUbbt/Xauthority", "HOME=/home/runner", "JULIA_PKG_PRECOMPILE_AUTO=0", "CONDARC=/tmp/jl_aWu15K/conda/3/x86_64/condarc-julia.yml", "JULIA_DEPOT_PATH=/tmp/jl_aWu15K", "OPENBLAS_MAIN_FREE=1", "R_HOME=*", "PYTHONIOENCODING=UTF-8"]), ProcessExited(1)) [1]

I tried to reproduce the error locally to understand the logic. And I found that it should be related to the building order of Conda.jl, RCall.jl, PyCall.jl. Here are my testing results for different building order

(I have deleted ~/.julia/conda before each attempt to reinstall conda, since the culprit is the conda version)

Specifically, after checking the build.log after building RCall.jl or PyCall.jl, both first install conda=23.1.0, and then update to be conda=23.5.0

The following packages will be UPDATED:

  conda                              23.1.0-py310hff52083_0 --> 23.5.0-py310hff52083_1

I am quite confused why the order makes difference!

Although the simple workaround seems be to build RCall before PyCall, I am still trying to figure out how to implement it in the AutoMerge automated task.

(I noticed that building conda actually did not install conda, conda is installed until building PyCall or RCall.)

Can anyone give some advices/hints? Thanks. Or is there any simple way to avoid the conflict conda versions?

szcf-weiya commented 1 year ago

Just notice that here is an option auto_update_conda for conda https://docs.conda.io/projects/conda/en/latest/configuration.html

By default it is True, so the above 23.1.0 would always upgrade to 23.5.0. And the error ResolvePackageNotFound: - conda==23.1.0 occurs after upgrading to 23.5.0 (although no idea about the logic why it only failed when building RCall after PyCall)

Anyway, one workaround is to disable the auto-update of conda. So before building PyCall, I manually add the following line

auto_update_conda: false

to condarc-julia.yml, and then build PyCall and RCall. It works!!!

However, I do not know how I can insert such config before building PyCall and RCall in the AutoMerge task. Since those building procedures are before my package, I cannot configure them before their builds. Any ideas? Thanks!

schlichtanders commented 1 year ago

thank you so much for filing this issue - your suggestion to disable auto_update_conda was working for me where other workarounds recently failed.

I shared my docker configuration here - maybe you can adapt it for your usecase https://github.com/JuliaPy/Conda.jl/issues/238#issuecomment-1627742313

szcf-weiya commented 1 year ago

@schlichtanders thanks for your feedback and share! Just pulled a request.