[BUG] Errors when building PyCall.jl

[MilesCranmer]

tldr, delete the folders:

• ~/.julia/packages/PyCall
• ~/.julia/prefs/PyCall
• ~/.julia/environments/pysr-{your-pysr-version}

and re-run python -m pysr install.

---

I am documenting this error because it has shown up a lot for people, and I want to document the solution.

The error is basically that after upgrading Python or Julia or both, and running PySR or re-installing PySR, you get the following error:

[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed but not compatible with this Python
└ executable. Re-building PyCall...
[ Info: Run `Pkg.build(“PyCall”; verbose=true)`
ERROR: LoadError: The following package names could not be resolved:
 * PyCall (not found in project or manifest)

The solution to this is to delete your ~/.julia/packages/PyCall folder. Then, simply re-run python -m pysr install.

If this gives you further errors in building, you should also delete ~/.julia/prefs/PyCall, and re-run the install step. For further issues, you could delete ~/.julia/environments folder (or more specifically, the folders ~/.julia/environments/pysr-{version} and ~/.julia/environments/{julia_version}, if you want to keep your other environments).

---

This should fix the problem. This issue is basically that the old installed version of PyCall.jl is attached to a specific combination of Julia and Python. And, for whatever reason, re-running julia.install() doesn't update the built version of PyCall.jl.

This is a temporary manual solution. In the future PySR should handle this automatically.

[MilesCranmer]

It might be good to automatically reset ~/.julia/prefs/PyCall whenever julia.install() is called? What do you think @mkitti?

[MilesCranmer]

I think PySR should catch julia.tools.PyCallInstallError and print out some PySR-relevant information. Especially the above debugging tips.

I think another couple of changes would simplify the install process for the user:

1. Have the PySR environment include the hash of the python path, so that using a different python would result in a new PySR install.
2. Have the PyCall.jl variables overwritten if PyCall is already installed in, e.g., @v1.8. Right now it seems like install PyCall.jl globally will just freeze the python path.

[mkitti]

Couldn't we just check for a mismatch between PyCall.python and whatever the current Python executable is? How do you find the latter?

[MilesCranmer]

I'm not sure, because the Python executable could have the same name. So ideally we would want to also store the version of Python and any other meta information that might affect things.

[MilesCranmer]

Also, a note to myself: should add unit tests for issues like #287 and #222. Would probably need to perform the test within a docker container with pyenv installed, and switch pyenv partway.

[mkitti]

We need more diagnostics here.

Can you catch this error and do one of the following?

println(stderr, Base.active_project())
println(stderr, read(Base.active_project()), String))
manifest = joinpath(dirname(Base.active_project()), "Manifest.toml"))
if isfile(manifest)
    println(stderr, read(manifest, String))
end

Somehow either Project.toml or Manifest.toml had been corrupted or the active environment is not what we think it is.

If there is something to delete, it might be the deps folder within thr package.

[MilesCranmer]

I'm trying to get a docker container that can consistently reproduce this.

FYI I've only seen the error when building via PyJulia; I'm not sure how to trigger it with PyCall directly.

[MilesCranmer]

My current attempt to generate this error. No error reported yet...

I wonder if somehow the latest PyCall has made this error go away.

Dockerfile:

```dockerfile FROM debian:bullseye-slim ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y curl git build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev libncurses5-dev libncursesw5-dev \ xz-utils libffi-dev liblzma-dev # Install juliaup: RUN curl -fsSL https://install.julialang.org | sh -s -- -y # Install pyenv: RUN curl -fsSL curl https://pyenv.run | sh && \ echo 'export PATH="/root/.pyenv/bin:$PATH"' >> ~/.bashrc && \ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \ echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc # Default to using bash -l: SHELL ["/bin/bash", "-l", "-c"] RUN juliaup add 1.8 && juliaup default 1.8 RUN pyenv install 3.9.2 && pyenv global 3.9.2 RUN python3 -m pip install --upgrade pip # Try to install pysr: RUN python3 -m pip install pysr RUN python3 -m pysr install # Change Julia and Python versions: RUN juliaup add 1.9 && juliaup default 1.9 && juliaup remove 1.8 RUN pyenv install 3.10 && pyenv global 3.10 RUN python3 -m pip install --upgrade pip # Try to re-install pysr: RUN python3 -m pip install pysr RUN python3 -m pysr install ```

---

One potential solution if I can end up producing the bug: make the PySR environment equal to pysr-{pysr_version}-jl-{julia_version}-py-{python_version}. That way the user would be forced to re-install PySR whenever any of those versions changes.

[MilesCranmer]

Okay I have something to reliably generate an error. If you install PySR with a specific Python version, and then change the Python version, remove ~/.julia/environments/pysr-0.14.2, and try to re-install PySR, you will get an error during precompilation due to a missing libpython file (it tries to load an older one):

Dockerfile:

```dockerfile FROM debian:bullseye-slim ENV DEBIAN_FRONTEND=noninteractive # Install juliaup and pyenv: RUN apt-get update && apt-get install -y curl # Install juliaup: RUN curl -fsSL https://install.julialang.org | sh -s -- -y RUN apt-get install -y git build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev libncurses5-dev libncursesw5-dev \ xz-utils libffi-dev liblzma-dev # Install pyenv: RUN curl -fsSL curl https://pyenv.run | sh && \ echo 'export PATH="/root/.pyenv/bin:$PATH"' >> ~/.bashrc && \ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \ echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc # Default to using bash -l: SHELL ["/bin/bash", "-l", "-c"] RUN juliaup add 1.8 && juliaup default 1.8 RUN pyenv install 3.9.2 && pyenv global 3.9.2 RUN python3 -m pip install --upgrade pip # Try to install pysr: RUN python3 -m pip install pysr==0.14.2 RUN python3 -m pysr install # Change Python version: RUN pyenv install 3.10 && pyenv global 3.10 && pyenv uninstall -f 3.9.2 RUN python3 -m pip install --upgrade pip # Try to use PySR: RUN python3 -m pip install pysr==0.14.2 RUN rm -r ~/.julia/environments/pysr-0.14.2 RUN python3 -m pysr install ```

This generates the following error during the `python -m pysr install` step:

``` #18 4.067 Precompiling project... #18 4.458 ✓ CompilerSupportLibraries_jll #18 4.551 ✓ VersionParsing #18 4.596 ✓ Preferences #18 4.899 ✓ PrecompileTools #18 5.654 ✓ MacroTools #18 19.31 ✓ Parsers #18 20.20 ✓ JSON #18 20.63 ✓ Conda #18 21.59 ✗ PyCall #18 21.61 8 dependencies successfully precompiled in 18 seconds. 3 already precompiled. #18 21.61 1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package #18 21.66 #18 21.66 Precompiling PyCall... #18 22.35 ERROR: LoadError: could not load library "/root/.pyenv/versions/3.9.2/lib/libpython3.9.so.1.0" #18 22.71 /root/.pyenv/versions/3.9.2/lib/libpython3.9.so.1.0: cannot open shared object file: No such file or directory. Please run `Pkg.build("PyCall")` if your Python build has changed #18 22.71 Stacktrace: #18 22.71 [1] error(::String, ::String) #18 22.72 @ Base ./error.jl:44 #18 22.72 [2] top-level scope #18 22.72 @ ~/.julia/packages/PyCall/ilqDX/src/startup.jl:51 #18 22.72 [3] include(mod::Module, _path::String) #18 22.72 @ Base ./Base.jl:457 #18 22.72 [4] include(x::String) #18 22.72 @ PyCall ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:1 #18 22.72 [5] top-level scope #18 22.72 @ ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:38 #18 22.72 [6] include #18 22.72 @ ./Base.jl:457 [inlined] #18 22.72 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) #18 22.72 @ Base ./loading.jl:2045 #18 22.72 [8] top-level scope #18 22.72 @ stdin:3 #18 22.72 in expression starting at /root/.julia/packages/PyCall/ilqDX/src/startup.jl:41 #18 22.72 in expression starting at /root/.julia/packages/PyCall/ilqDX/src/PyCall.jl:1 #18 22.72 in expression starting at stdin:3 #18 22.72 #18 22.72 caused by: could not load library "/root/.pyenv/versions/3.9.2/lib/libpython3.9.so.1.0" #18 22.72 /root/.pyenv/versions/3.9.2/lib/libpython3.9.so.1.0: cannot open shared object file: No such file or directory #18 22.72 Stacktrace: #18 22.72 [1] dlopen(s::String, flags::UInt32; throw_error::Bool) #18 22.72 @ Base.Libc.Libdl ./libdl.jl:117 #18 22.72 [2] dlopen(s::String, flags::UInt32) #18 22.72 @ Base.Libc.Libdl ./libdl.jl:116 #18 22.72 [3] top-level scope #18 22.72 @ ~/.julia/packages/PyCall/ilqDX/src/startup.jl:48 #18 22.72 [4] include(mod::Module, _path::String) #18 22.72 @ Base ./Base.jl:457 #18 22.72 [5] include(x::String) #18 22.72 @ PyCall ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:1 #18 22.72 [6] top-level scope #18 22.72 @ ~/.julia/packages/PyCall/ilqDX/src/PyCall.jl:38 #18 22.72 [7] include #18 22.72 @ ./Base.jl:457 [inlined] #18 22.72 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) #18 22.73 @ Base ./loading.jl:2045 #18 22.73 [9] top-level scope #18 22.73 @ stdin:3 #18 22.89 ERROR: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to "/root/.julia/compiled/v1.9/PyCall/jl_RA5NCO". #18 23.47 Stacktrace: #18 23.47 [1] error(s::String) #18 23.48 @ Base ./error.jl:35 #18 23.48 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool) #18 23.48 @ Base ./loading.jl:2296 #18 23.48 [3] compilecache #18 23.48 @ ./loading.jl:2163 [inlined] #18 23.48 [4] _require(pkg::Base.PkgId, env::String) #18 23.48 @ Base ./loading.jl:1805 #18 23.48 [5] _require_prelocked(uuidkey::Base.PkgId, env::String) #18 23.48 @ Base ./loading.jl:1660 #18 23.48 [6] macro expansion #18 23.48 @ ./loading.jl:1648 [inlined] #18 23.48 [7] macro expansion #18 23.48 @ ./lock.jl:267 [inlined] #18 23.48 [8] require(into::Module, mod::Symbol) #18 23.48 @ Base ./loading.jl:1611 #18 23.52 Traceback (most recent call last): #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/runpy.py", line 196, in _run_module_as_main #18 23.52 return _run_code(code, main_globals, None, #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/runpy.py", line 86, in _run_code #18 23.52 exec(code, run_globals) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/pysr/__main__.py", line 4, in #18 23.52 _cli(prog_name="pysr") #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ #18 23.52 return self.main(*args, **kwargs) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/click/core.py", line 1055, in main #18 23.52 rv = self.invoke(ctx) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/click/core.py", line 1657, in invoke #18 23.52 return _process_result(sub_ctx.command.invoke(sub_ctx)) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/click/core.py", line 1404, in invoke #18 23.52 return ctx.invoke(self.callback, **ctx.params) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/click/core.py", line 760, in invoke #18 23.52 return __callback(*args, **kwargs) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/pysr/_cli/main.py", line 37, in _install #18 23.52 install(julia_project, quiet, precompile) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/pysr/julia_helpers.py", line 84, in install #18 23.52 julia.install(quiet=quiet) #18 23.52 File "/root/.pyenv/versions/3.10.12/lib/python3.10/site-packages/julia/tools.py", line 127, in install #18 23.52 raise PyCallInstallError("Precompiling") #18 23.52 julia.tools.PyCallInstallError: Precompiling PyCall failed. #18 23.52 #18 23.52 ** Important information from Julia may be printed before Python's Traceback ** #18 23.52 #18 23.52 Some useful information may also be stored in the build log file #18 23.52 `~/.julia/packages/PyCall/*/deps/build.log`. #18 23.52 ------ ```

There should definitely be a way we can automatically fix this, right? Is it prefs we need to update?

---

I note that even doing julia -e 'using Pkg; Pkg.add("PyCall"); Pkg.build("PyCall")' beforehand does not seem to save things. The python -m pysr install still fails...

[MilesCranmer]

It seems like PyCall.jl is storing the libpython version inside packages/PyCall/ilqDX/deps/deps.jl:

const libpython = "/root/.pyenv/versions/3.9.2/lib/libpython3.9.so.1.0"

Maybe that's the issue? Perhaps a new package directory is not being created when the Python binary changes?

[mkitti]

Run a Julia process that executes this before trying to connect Julia and Python.

using Pkg
Pkg.activate("pysr..."; shared=true)
ENV["PYTHON"] = "/path/to/python"
Pkg.build("PyCall")

That will force the following file to run again.

https://github.com/JuliaPy/PyCall.jl/blob/master/deps/build.jl

[MilesCranmer]

That's a good idea. Let me try that.

[MilesCranmer]

I implemented your suggestion in #363 so it automatically updates PyCall. Seems to work locally! I added a unittest too.

[MilesCranmer]

Okay I'm temporarily going to close this with #363 merged. Fingers crossed. I'll leave it pinned though.

[MilesCranmer]

Fixed by #535