[BUG] Installation issue: `const PyCall = Base.require`

[flxmr]

Describe the bug I am using Python installed in a spack environment and want to use PySR (the following works with the system-python). So I installed Julia, added it to my PATH and then I do the installation instructions, which fail:

>>> import pysr
>>> pysr.install()
[ Info: Julia version info
Julia Version 1.8.5
< all the diagnostics >
┌ Info: PyCall is already installed and compatible with Python executable.
│ 
│ PyCall:
│     python: /software_modules/spack/var/spack/environments/ml-23-03/.spack-env/view/bin/python
│     libpython: /software_modules/spack/opt/spack/linux-fedora36-zen/gcc-8.5.0/python-3.10.10-ldbbqz6242q36jrznozylpqzburhgu2v/lib/libpython3.10.so.1.0
│ Python:
│     python: /software_modules/spack/var/spack/environments/ml-23-03/.spack-env/view/bin/python
└     libpython: /software_modules/spack/opt/spack/linux-fedora36-zen/gcc-8.5.0/python-3.10.10-ldbbqz6242q36jrznozylpqzburhgu2v/lib/libpython3.10.so.1.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.local/lib/python3.10/site-packages/pysr/julia_helpers.py", line 85, in install
    Main, init_log = init_julia(julia_project, quiet=quiet, return_aux=True)
  File "/home/.local/lib/python3.10/site-packages/pysr/julia_helpers.py", line 192, in init_julia
    Julia(**julia_kwargs)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 519, in __init__
    self._call("const PyCall = Base.require({0})".format(PYCALL_PKGID))
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 555, in _call
    self.check_exception(src)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'InitError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))

Curiously it works calling the julia-library before:

Python 3.10.10 (main, Mar 31 2023, 22:33:35) [GCC 8.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import julia
>>> julia.Julia()
<julia.core.LegacyJulia object at 0x7f7fff0a0550>
>>> import pysr
>>> pysr.install()

(I can run the example in this REPL)

Closing that REPL and trying to run the example again, I arrive at:

>>> model.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.local/lib/python3.10/site-packages/pysr/sr.py", line 1834, in fit
    self._run(X, y, mutated_params, weights=weights, seed=seed)
  File "/home/.local/lib/python3.10/site-packages/pysr/sr.py", line 1525, in _run
    Main = init_julia(self.julia_project, julia_kwargs=julia_kwargs)
  File "/home/.local/lib/python3.10/site-packages/pysr/julia_helpers.py", line 192, in init_julia
    Julia(**julia_kwargs)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 519, in __init__
    self._call("const PyCall = Base.require({0})".format(PYCALL_PKGID))
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 555, in _call
    self.check_exception(src)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'InitError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))

This is weird, do you have any idea what could be the problem? Should I raise this with the Pycall-Developers?

Version (please include the following information):

• OS: Linux
• Julia 1.8.5
• Python 3.10.10
• pip-install
• PySR 0.12.1

[MilesCranmer]

One thing to try, which is a bit of a hack but might fix it — manually install and build PyCall in the base Julia environment:

julia -e 'using Pkg; Pkg.install("PyCall"); Pkg.build("PyCall")'

Run this line in bash in any directory, doesn’t matter which. (Make sure to have the same Python available on your path as if you were using pysr.install())

This should add PyCall to the base env which might fix this. Let me know if this works. If not I would then try the solutions in https://github.com/MilesCranmer/PySR/issues/257

Cheers, Miles

[dbl001]

Perhaps this is the same issue ... I can run the Feynman problems from iPython in a terminal, however, I get this error running the code from inside 'Pycharm'

/Users/davidlaxer/anaconda3/envs/pysr_test/bin/python /Users/davidlaxer/PySR/test.py 
DEBUG (28160) 
DEBUG (28160) Debug-level logging is enabled for PyJulia.
DEBUG (28160) PyJulia version: 0.6.0
DEBUG (28160) 
/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/juliainfo.py:93: UserWarning: julia warned:
The latest version of Julia in the `1.9` channel is 1.9.0-rc3+0.x64.apple.darwin14. You currently have `1.9.0-rc2+0.x64.apple.darwin14` installed. Run:

  juliaup update

to install Julia 1.9.0-rc3+0.x64.apple.darwin14 and update the `1.9` channel to that version.
  warnings.warn("{} warned:\n{}".format(julia, stderr))
DEBUG (28160) pyprogramname = None
DEBUG (28160) sys.executable = /Users/davidlaxer/anaconda3/envs/pysr_test/bin/python
DEBUG (28160) bindir = /Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/bin
DEBUG (28160) libjulia_path = /Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/lib/libjulia.1.9.dylib
DEBUG (28160) is_compatible_python = None
DEBUG (28160) use_custom_sysimage = False
DEBUG (28160) compiled_modules = 'no'
DEBUG (28160) argv_list = [b'/Users/davidlaxer/anaconda3/envs/pysr_test/bin/python', b'--compiled-modules=no', b'--threads=auto']
DEBUG (28160) argc = c_int(3)
DEBUG (28160) jl_parse_opts called
DEBUG (28160) argc = c_int(0)
DEBUG (28160) calling jl_init_with_image(/Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/bin, /Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/lib/julia/sys.dylib)
DEBUG (28160) seems to work...
┌ Error: Failed to import PyCall
│   exception =
│    ArgumentError: Package PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] is required but does not seem to be installed:
│     - Run `Pkg.instantiate()` to install all recorded dependencies.
│    
│    Stacktrace:
│     [1] _require(pkg::Base.PkgId, env::Nothing)
│       @ Base ./loading.jl:1741
│     [2] _require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│       @ Base ./loading.jl:1623
│     [3] _require_prelocked
│       @ ./loading.jl:1621 [inlined]
│     [4] macro expansion
│       @ ./lock.jl:267 [inlined]
│     [5] require(uuidkey::Base.PkgId)
│       @ Base ./loading.jl:1618
│     [6] top-level scope
│       @ none:3
└ @ Main none:5
DEBUG (28160) exception occured? 5841618960
Traceback (most recent call last):
  File "/Users/davidlaxer/PySR/test.py", line 11, in <module>
    julia = Julia(compiled_modules=False, debug=True, threads='auto')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 702, in __init__
    self.__julia = Julia(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 510, in __init__
    self._call("""
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 555, in _call
    self.check_exception(src)
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'ArgumentError' occurred while calling julia code:

            const PyCall = try
                Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))
            catch err
                @error "Failed to import PyCall" exception = (err, catch_backtrace())
                rethrow()
            end

SYSTEM: caught exception of type :ArgumentError while trying to print a failed Task notice; giving up

Process finished with exit code 1

Here's the test script:

import os

# Set the number of threads for Julia
os.environ["JULIA_NUM_THREADS"] = "4"

import pysr

from julia import Julia
julia = Julia(compiled_modules=False, debug=True, threads='auto')

from julia import Main
from julia.tools import redirect_output_streams

#redirect_output_streams()

pysr.install()

from pysr.sr import pysr, best
from pysr.feynman_problems import FeynmanProblem
from pysr.feynman_problems import mk_problems
from pysr.feynman_problems import run_on_problem
from pysr.feynman_problems import do_feynman_experiments
from pysr.feynman_problems import do_feynman_experiments_parallel

import sympy
from sympy.core.random import _randint
import numpy as np
from matplotlib import pyplot as plt
from pysr import PySRRegressor
import torch
from torch import nn, optim
from torch.nn import functional as F
from torch.utils.data import DataLoader, TensorDataset
import pytorch_lightning as pl
from sklearn.model_selection import train_test_split

do_feynman_experiments()

Environment:

% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-rc3 (2023-04-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pkg

julia> Pkg.add("PyCall")
   Resolving package versions...
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Project.toml`
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Manifest.toml`

julia> Pkg.update("PyCall")
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Project.toml`
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Manifest.toml`

julia> 
(pysr_test) davidlaxer@bluediamond julia % ipython          
Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:51) [Clang 14.0.4 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from julia.api import JuliaInfo

In [2]: info = JuliaInfo.load()

In [3]: info.is_compatible_python()
Out[3]: False

In [4]: info.python
Out[4]: '/Users/davidlaxer/anaconda3/envs/pysr_test/bin/python'

In [5]: info.sysimage
Out[5]: '/Users/davidlaxer/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/lib/julia/sys.dylib'

[dbl001]

In my environment, PyCharm was invoking julia in:

/Users/davidlaxer/.julia/juliaup/julia-1.9.0+0.x64.apple.darwin14/bin/julia

instead of from my conda virtual environment:

/Users/davidlaxer/anaconda3/envs/pysr_test//bin/julia

The julia environment in: /Users/davidlaxer/.julia/juliaup/julia-1.9.0+0.x64.apple.darwin14/bin/julia did not have 'pycall' installed.

[MilesCranmer]

Hm, this might be a different issue. Did you install PySR with pip instead of conda?

[dbl001]

Look like: $ python setup.py develop

% conda list pysr
# packages in environment at /Users/davidlaxer/anaconda3/envs/pysr_test:
#
# Name                    Version                   Build  Channel
pysr                      0.12.3                    dev_0    <develop>
(pysr_test) davidlaxer@bluediamond julia % pip show pysr
Name: pysr
Version: 0.11.11
Summary: Simple and efficient symbolic regression
Home-page: https://github.com/MilesCranmer/pysr
Author: Miles Cranmer
Author-email: miles.cranmer@gmail.com
License: 
Location: /Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages
Requires: julia, numpy, pandas, scikit-learn, sympy
Required-by: 

[MilesCranmer]

Okay I think that's a different issue. If using conda's Julia then you likely would also need to use conda's PySR (https://github.com/conda-forge/pysr-feedstock) so it gets the paths correctly.

[MilesCranmer]

Presumably fixed by #535, so closing.