ResolvePackageNotFound: - conda==23.1.0

[schlichtanders]

Hi, I am using Conda.jl and it failed on the most recent Julia 1.9.0 docker image to install packages

I explicitly set the environment variable PYTHON="" to force using Conda.jl default conda installation. You can for instance start the docker with

docker run -it --rm julia:1.9 julia

then run:

import Pkg
Pkg.add("Conda")
using Conda
Conda.add("numpy")
Conda.add("matplotlib")  # the other way around it works, i.e. first installing matplotlib and then numpy

Which should output something like

[ Info: Running `conda install -q -y numpy pandas matplotlib plotly` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.

ResolvePackageNotFound: 
  - conda==23.1.0

ERROR: LoadError: failed process: Process(setenv(`/github/home/.julia/conda/3/x86_64/bin/conda install -q -y numpy pandas matplotlib plotly`,["PATH=/usr/local/julia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "JOLIN_ENVIRONMENT=test", "GITHUB_RUN_NUMBER=16", "JULIA_PATH=/usr/local/julia", "GITHUB_REF_NAME=45/merge", "RUNNER_ARCH=X64", "GITHUB_REF=refs/pull/45/merge", "GITHUB_STEP_SUMMARY=/__w/_temp/_runner_file_commands/step_summary_ad717c8d-77e6-488e-8547-c929ccb04ec2", "GITHUB_REPOSITORY=jolin-io/JolinWorkspaceTemplate", "TZ=UTC\u2060"  …  "GITHUB_WORKFLOW=Test", "GITHUB_ACTION_REPOSITORY=", "GITHUB_REPOSITORY_OWNER=jolin-io", "GITHUB_EVENT_NAME=pull_request", "HOME=/github/home", "GITHUB_ACTION=__run_4", "OPENBLAS_MAIN_FREE=1", "HOSTNAME=0051195642c0", "PYTHONIOENCODING=UTF-8", "GITHUB_EVENT_PATH=/github/workflow/event.json"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run
   @ ./process.jl:477 [inlined]
 [4] runconda(args::Cmd, env::String)
   @ Conda ~/.julia/packages/Conda/kOnIE/src/Conda.jl:128
 [5] add(pkg::Vector{String}, env::String; channel::String)
   @ Conda ~/.julia/packages/Conda/kOnIE/src/Conda.jl:224
 [6] add
   @ ~/.julia/packages/Conda/kOnIE/src/Conda.jl:222 [inlined]
 [7] add(pkg::Vector{String})
   @ Conda ~/.julia/packages/Conda/kOnIE/src/Conda.jl:222

There is also a related discussion on discourse which states the workaround to preinstall conda with conda==23.1.0. But this is only a workaround.

EDIT: there is another more recent discourse thread which also mentions this problem and had a different workaround.

[schlichtanders]

This seems to be amd64 specific. When running it on arm64, it works without problems.

[schlichtanders]

My workaround recently failed, hence looked again into this issue.

It seems the underlying problem is that PyCall will install "numpy" first. Doing

Conda.add("numpy")
Conda.add("matplotlib")

will fail while the opposite

Conda.add("matplotlib")
Conda.add("numpy")

works without problems.

Unfortunately everyone who uses PyCall will first install numpy and then (maybe) matplotlib. I updated the issue summary respectively.

[schlichtanders]

@stevengj can you take a look?

[schlichtanders]

Found a new workaround by this similar issue https://github.com/JuliaPy/Conda.jl/issues/242, i.e. setting auto_update_conda: false in the condarc-julia.yml file. (must be created top level inside the conda root environment, and should be initialized with the content of the existing .condarc file next to it).

Here my dockerfile lines which fix Conda.jl successfully for me

# Setting up Conda, Python and R
# this is needed because of a bug https://github.com/JuliaPy/Conda.jl/issues/238
# for the cleanup steps I followed https://jcristharif.com/conda-docker-tips.html
RUN wget -O Miniforge.sh "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" \
    && bash Miniforge.sh -b -p "${USER_HOME_DIR}/conda" \
    && rm Miniforge.sh \
    && source "${USER_HOME_DIR}/conda/etc/profile.d/conda.sh" \
    && cat "${USER_HOME_DIR}/conda/.condarc" > "${USER_HOME_DIR}/conda/condarc-julia.yml" \
    && echo "auto_update_conda: false" >> "${USER_HOME_DIR}/conda/condarc-julia.yml" \
    && conda clean -afy \
    && find ./conda/ -follow -type f -name '*.a' -delete \
    && find ./conda/ -follow -type f -name '*.pyc' -delete \
    && find ./conda/ -follow -type f -name '*.js.map' -delete

ENV PATH="${USER_HOME_DIR}/conda/bin:${PATH}"
ENV CONDA_JL_CONDA_EXE="${USER_HOME_DIR}/conda/bin/conda"
ENV CONDA_JL_HOME="${USER_HOME_DIR}/conda"
# force PyCall.jl use Conda.jl
ENV PYTHON=""