search

MilesCranmer / PySR

High-Performance Symbolic Regression in Python and Julia
https://astroautomata.com/PySR
Apache License 2.0
2.35k stars 211 forks source link

[BUG] Can't re-build PyCall.jl after upgrading Python #222

Closed MilesCranmer closed 1 year ago

MilesCranmer commented 1 year ago

Just documenting this in case others encounter it.

This is a weird bug often reported on PyCall.jl where PyCall will somehow cache the location of your old Python binary, and re-building will fail.

I downgraded my Python version from 3.11 to 3.10, and tried to do python -c 'import pysr; pysr.install()'. This gave me the error:

┌ Info: PyCall is already installed but not compatible with this Python
└ executable.  Re-building PyCall...
[ Info: Run `Pkg.build("PyCall"; verbose=true)`
ERROR: LoadError: The following package names could not be resolved:
 * PyCall (not found in project or manifest)
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] ensure_resolved(ctx::Pkg.Types.Context, manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
   @ Pkg.Types ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:952
 [3] ensure_resolved
   @ ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:904 [inlined]
 [4] build(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; verbose::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
   @ Pkg.API ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:1028
 [5] build(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Bool}}})
   @ Pkg.API ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:156
 [6] #build#99
   @ ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:144 [inlined]
 [7] #build#98
   @ ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:143 [inlined]
 [8] build_pycall()
   @ Main ~/venvs/main/lib/python3.10/site-packages/julia/install.jl:54
 [9] top-level scope
   @ ~/venvs/main/lib/python3.10/site-packages/julia/install.jl:108
in expression starting at /Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/install.jl:73
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/mcranmer/Documents/PySR/pysr/julia_helpers.py", line 79, in install
    julia.install(quiet=quiet)
  File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/tools.py", line 118, in install
    raise PyCallInstallError("Installing", output)
julia.tools.PyCallInstallError: Installing PyCall failed.

** Important information from Julia may be printed before Python's Traceback **

Some useful information may also be stored in the build log file
`~/.julia/packages/PyCall/*/deps/build.log`.

If I open up Julia, install PyCall.jl into any environment, and write:

julia> using PyCall

julia> PyCall.python
"/Users/mcranmer/venvs/main/bin/python3.11"

it gives me the old location of the Python binary! The only one in that folder is now python3.10 - the python3.11 was removed. This is even true if I uninstall and re-install PyCall.jl. It will somehow remember the old location, and re-building it will fail. Maybe it expects PyCall.python to be unchanged between Python versions?

The way I ended up fixing this is to run (inside Julia)

using Pkg
ENV["PYTHON"] = "/Users/mcranmer/venvs/main/bin/python3.10"
pkg"build PyCall"

this successfully re-builds PyCall.jl. But I think ideally this shouldn't be necessary, to lower friction for users. To do this, I think PyJulia should check what Python binary was used to launch it, and, if necessary, reset ENV and rebuild.

FYI @mkitti @stevengj

mkitti commented 1 year ago

This is probably due to a stale deps.jl file. We should probably move to a Preferences.jl mechanism for this or again take inspiration from PythonCall.jl.

mkitti commented 1 year ago

You should be able to locate the deps file via this code.

using PyCall # repeat if this errors
readdir(joinpath(dirname(pathof(PyCall)), "..", "deps"))
MilesCranmer commented 1 year ago

Maybe this could be checked at julia.install(), and, if deps.jl's python key doesn't match the calling binary, it gets reset?

mkitti commented 1 year ago

This would be much easier to handle via Preferences.jl. We should use that.

dbl001 commented 1 year ago

Will 'PySr' run with Julia 1.9?

% ipython
Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:51) [Clang 14.0.4 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pysr

In [2]: pysr.install()
[ Info: Julia version info
Julia Version 1.9.0-rc3
Commit 1853b903282 (2023-04-26 15:51 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  uname: Darwin 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64 i386
  CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz: 
                 speed         user         nice          sys         idle          irq
       #1-16  3800 MHz    4265439 s          0 s     764858 s   16874798 s          0 s
  Memory: 128.0 GB (4417.48828125 MB free)
  Uptime: 274568.0 sec
  Load Avg:  7.181640625  7.73876953125  12.20849609375
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 16 virtual cores
Environment:
  JULIA_DEPOT_PATH_BACKUP = 
  JULIA_PROJECT_BACKUP = 
  JULIA_LOAD_PATH_BACKUP = 
  JULIA_DEPOT_PATH = /Users/davidlaxer/anaconda3/envs/pysr_test/share/julia:/Users/davidlaxer/anaconda3/envs/pysr_test/share/pysr/depot:
  JULIA_PROJECT = @pysr-0.12.3
  JULIA_LOAD_PATH = @:@pysr_test:@stdlib
  JULIA_SSL_CA_ROOTS_PATH_BACKUP = 
  JULIA_CONDAPKG_BACKEND_BACKUP = 
  JULIA_CONDAPKG_BACKEND = System
  JULIA_CONDAPKG_EXE_BACKUP = 
  JULIA_CONDAPKG_EXE = /Users/davidlaxer/anaconda3/bin/conda
  JULIA_DEPOT_PATH_PYSR_BACKUP = /Users/davidlaxer/anaconda3/envs/pysr_test/share/julia:
  TERM = xterm-256color
  PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/bin:/Users/davidlaxer/.juliaup/bin:/Users/davidlaxer/.cabal/bin:/Users/davidlaxer/.ghcup/bin:/Users/davidlaxer/anaconda3/envs/pysr_test/bin:/Users/davidlaxer/anaconda3/condabin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/davidlaxer/.cargo/bin:/Users/jetbrains/.local/bin
  XPC_FLAGS = 0x0
  HOME = /Users/davidlaxer
  JAVA_HOME = :-
  JAVA_LD_LIBRARY_PATH = :-
  CAML_LD_LIBRARY_PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/stublibs:/Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/ocaml/stublibs:/Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/ocaml
  OCAML_TOPLEVEL_PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/toplevel
  PKG_CONFIG_PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/pkgconfig:
  CONDA_BACKUP_CPPFLAGS = -D_FORTIFY_SOURCE=2 -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_CFLAGS = -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_LDFLAGS = -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/davidlaxer/anaconda3/lib -L/Users/davidlaxer/anaconda3/lib
  CONDA_BACKUP_LDFLAGS_LD = -pie -headerpad_max_install_names -dead_strip_dylibs -rpath /Users/davidlaxer/anaconda3/lib -L/Users/davidlaxer/anaconda3/lib
  CONDA_BACKUP_DEBUG_CFLAGS = -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -Og -g -Wall -Wextra -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_CMAKE_PREFIX_PATH = :/Users/davidlaxer/anaconda3
  CONDA_BACKUP_FFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_FORTRANFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_DEBUG_FFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_DEBUG_FORTRANFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_JL_HOME_BACKUP = 
  CONDA_JL_HOME = /Users/davidlaxer/anaconda3/envs/pysr_test
[ Info: Julia executable: /Users/davidlaxer/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/bin/julia
[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed but not compatible with this Python
└ executable.  Re-building PyCall...
[ Info: Run `Pkg.build("PyCall"; verbose=true)`
ERROR: LoadError: The following package names could not be resolved:
 * PyCall (not found in project or manifest)
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
  [2] ensure_resolved(ctx::Pkg.Types.Context, manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
    @ Pkg.Types ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Types.jl:995
  [3] ensure_resolved
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Types.jl:946 [inlined]
  [4] build(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; verbose::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:1052
  [5] build(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Bool}}})
    @ Pkg.API ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
  [6] #build#85
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
  [7] build
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
  [8] #build#84
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
  [9] build
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
 [10] build_pycall()
    @ Main ~/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/install.jl:54
 [11] top-level scope
    @ ~/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/install.jl:108
in expression starting at /Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/install.jl:73
---------------------------------------------------------------------------
PyCallInstallError                        Traceback (most recent call last)
Cell In[2], line 1
----> 1 pysr.install()

File ~/PySR/pysr/julia_helpers.py:84, in install(julia_project, quiet, precompile)
     81 if precompile == False:
     82     os.environ["JULIA_PKG_PRECOMPILE_AUTO"] = "0"
---> 84 julia.install(quiet=quiet)
     85 Main, init_log = init_julia(julia_project, quiet=quiet, return_aux=True)
     86 io_arg = _get_io_arg(quiet)

File ~/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/tools.py:118, in install(julia, color, python, quiet)
    116     return
    117 elif returncode != 0:
--> 118     raise PyCallInstallError("Installing", output)
    120 if not quiet:
    121     print(file=sys.stderr)

PyCallInstallError: Installing PyCall failed.

** Important information from Julia may be printed before Python's Traceback **

Some useful information may also be stored in the build log file
`~/.julia/packages/PyCall/*/deps/build.log`.

In [3]: quit
MilesCranmer commented 1 year ago

Yes, it works for me. Just follow the instructions here: https://github.com/MilesCranmer/PySR/issues/257

dbl001 commented 1 year ago

Works for me now. Thanks!

MilesCranmer commented 1 year ago

Cool. Yes I would really like to have that bug automatically solved by PySR, but I'm not sure how to do it so far.

MilesCranmer commented 1 year ago

Self-repair added by #363, so closing this. (Will re-open if any other issues)