search

MilesCranmer / PySR

High-Performance Symbolic Regression in Python and Julia
https://astroautomata.com/PySR
Apache License 2.0
2.32k stars 211 forks source link

[BUG]: Julia interface fails on conda environments #448

Closed IlyaOrson closed 11 months ago

IlyaOrson commented 11 months ago

What happened?

I get the following error with a fresh install in a fresh conda environment with pysr installed with pip.

ImportError: cannot import name 'Main' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr_std\Lib\site-packages\julia\__init__.py)

Version

0.16.3

Operating System

Windows

Package Manager

Other (specify below)

Interface

IPython Terminal

Relevant log output

>>> model.fit(X, y)
Compiling Julia backend...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ilyao\miniforge3\envs\pysr_std\Lib\site-packages\pysr\sr.py", line 1970, in fit
    self._run(X, y, mutated_params, weights=weights, seed=seed)
  File "C:\Users\ilyao\miniforge3\envs\pysr_std\Lib\site-packages\pysr\sr.py", line 1625, in _run
    Main = init_julia(self.julia_project, julia_kwargs=julia_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_std\Lib\site-packages\pysr\julia_helpers.py", line 216, in init_julia
    from julia import Main as _Main
ImportError: cannot import name 'Main' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr_std\Lib\site-packages\julia\__init__.py)

Extra Info

My julia version is 1.9.3, installed with juliaup.

In the fresh environment I can get pyjulia working but only with the lower level interface

import julia
julia.core.Julia() # works
from julia import Main #fails
MilesCranmer commented 11 months ago

Did you run the python -m pysr install step? Any issues from it?

MilesCranmer commented 11 months ago

Also what is your PyJulia version?

IlyaOrson commented 11 months ago

I should have specified that I am trying to use a local clone of the back end. My PyJulia version is 0.6.1.

I built PyCall manually within the local SymbolicRegression.jl project so that that it points to the python within the pysr conda environment. When I run import pysr; pysr.install(julia_project) the output looks fine:

...
JULIA_PROJECT = D:\ilyao\Documents\research\coding\physics_informed_ADoK\SymbolicRegression.jl
[ Info: Julia executable: C:\Users\ilyao\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\bin\julia.exe
[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed and compatible with Python executable.
│
│ PyCall:
│     python: C:\Users\ilyao\miniforge3\envs\pysr\python.exe
│     libpython: C:\Users\ilyao\miniforge3\envs\pysr\python312.dll
│ Python:
│     python: C:\Users\ilyao\miniforge3\envs\pysr\python.exe
└     libpython: C:\Users\ilyao\miniforge3\envs\pysr\python312.dll
  Activating project at `D:\ilyao\Documents\research\coding\physics_informed_ADoK\SymbolicRegression.jl`
Precompiling project...
  11 dependencies successfully precompiled in 91 seconds. 75 already precompiled.

I can run the regression directly on the local backend from pyjulia as well:

import julia
j = julia.core.Julia()
j.eval("Base.active_project()")  # 'D:\\ilyao\\Documents\\research\\coding\\physics_informed_ADoK\\SymbolicRegression.jl\\Project.toml'
j.eval("""
import SymbolicRegression: Options, equation_search

X = randn(2, 100)
y = 2 * cos.(X[2, :]) + X[1, :] .^ 2 .- 2

options = Options(
    binary_operators=[+, *, /, -],
    unary_operators=[cos, exp],
    populations=20
)

hall_of_fame = equation_search(
    X, y, niterations=40, options=options,
    parallelism=:multithreading
)  # works!
"""
)

But I cannot import any module directly from pyjulia

from julia import SymbolicRegression  # (Base and Main also fail)
ImportError: cannot import name 'SymbolicRegression' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr\Lib\site-packages\julia\__init__.py)
MilesCranmer commented 11 months ago

Can you instead try with

from pysr.julia_helpers import init_julia
init_julia(julia_project="/path/to/your/julia/project")

and then from julia import Main; from julia import SymbolicRegression?

IlyaOrson commented 11 months ago

I get the same error

In [1]: from pysr.julia_helpers import init_julia

In [2]: julia_project = "D:\\ilyao\\Documents\\research\\coding\\physics_informed_ADoK\\SymbolicRegression.jl"

In [3]: init_julia(julia_project)
Out[3]: <julia.core.Julia at 0x1f20a2ec860>

In [4]: from julia import Main
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[4], line 1
----> 1 from julia import Main

ImportError: cannot import name 'Main' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr\Lib\site-packages\julia\__init__.py)
MilesCranmer commented 11 months ago

That’s weird. This is an issue I’ve never seen before.

I guess this is PyJulia-specific as there’s no PySR-specific parts. Maybe check if there are any related issues on the PyJulia side? Also there is this troubleshooting guide: https://pyjulia.readthedocs.io/en/stable/troubleshooting.html#

Btw, does it work outside of the custom environment? Like just in a normal PySR env (run the quickstart example)? I don’t quite understand how it seems to correctly select the custom project if you just run Julia(), are you running anything before that?

MilesCranmer commented 11 months ago

Also you mentioned you are using PySR installed with pip inside a conda env. Is PyJulia also installed with pip?

IlyaOrson commented 11 months ago

Pip installed PyJulia as well while installing pysr. I avoided using conda to install pysr to be able to get the latest version on windows, I am just using it for the environment management in python (I use mamba really but the environment management is done by conda afaik).

I just tried to install pysr without the custom backend on a fresh conda environment and I got the same error.

In a fresh conda environment I just add pip to make sure it is the one within the conda environment.

conda create -n pysr_mamba
conda activate pysr_mamba
mamba install pip -Y

python -m pysr install

[ Info: Julia version info
Julia Version 1.9.3
...
JULIA_PROJECT = @pysr-0.16.3
...
[ Info: Julia executable: C:\Users\ilyao\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\bin\julia.exe
[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed but not compatible with this Python
└ executable.  Re-building PyCall...
[ Info: Run `Pkg.build("PyCall"; verbose=true)`
    Building Conda ─→ `C:\Users\ilyao\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\8c86e48c0db1564a1d49548d3515ced5d604c408\build.log`
    Building PyCall → `C:\Users\ilyao\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\43d304ac6f0354755f1d60730ece8c499980f7ba\build.log`
[ Info: PyCall is using C:\Users\ilyao\miniforge3\envs\pysr_mamba\python.exe (Python 3.12.0) at C:\Users\ilyao\miniforge3\envs\pysr_mamba\python.exe, libpython = C:\Users\ilyao\miniforge3\envs\pysr_mamba\python312.dll
[ Info: C:\Users\ilyao\.julia\packages\PyCall\ilqDX\deps\deps.jl has been updated
[ Info: C:\Users\ilyao\.julia\prefs\PyCall has been updated
Precompiling project...
  ✓ PyCall
  1 dependency successfully precompiled in 18 seconds. 11 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
[ Info: `C:\Users\ilyao\.julia\packages\PyCall\ilqDX\src\..\deps\deps.jl`
const python = "C:\\Users\\ilyao\\miniforge3\\envs\\pysr_mamba\\python.exe"
const libpython = "C:\\Users\\ilyao\\miniforge3\\envs\\pysr_mamba\\python312.dll"
const pyprogramname = "C:\\Users\\ilyao\\miniforge3\\envs\\pysr_mamba\\python.exe"
const pyversion_build = v"3.12.0"
const PYTHONHOME = "C:\\Users\\ilyao\\miniforge3\\envs\\pysr_mamba"

"True if we are using the Python distribution in the Conda package."
const conda = false

Precompiling PyCall...
Precompiling PyCall... DONE
PyCall is installed and built successfully.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\pysr\__main__.py", line 4, in <module>
    _cli(prog_name="pysr")
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\pysr\_cli\main.py", line 37, in _install
    install(julia_project, quiet, precompile)
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\pysr\julia_helpers.py", line 98, in install
    Main, init_log = init_julia(julia_project, quiet=quiet, return_aux=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\pysr\julia_helpers.py", line 216, in init_julia
    from julia import Main as _Main
ImportError: cannot import name 'Main' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\julia\__init__.py)

If I try to use the lower-level pyjulia interface I get the following error:

>>> import julia
>>> j = julia.core.Julia()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\julia\core.py", line 489, in __init__
    raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.

Julia executable:
    julia
Python interpreter and libpython used by PyCall.jl:
    None
    None
Python interpreter used to import PyJulia and its libpython.
    C:\Users\ilyao\miniforge3\envs\pysr_mamba\python.exe
    C:\Users\ilyao\miniforge3\envs\pysr_mamba\python312.dll

In Julia >= 0.7, above two paths to `libpython` have to match exactly
in order for PyJulia to work out-of-the-box.  To configure PyCall.jl to use
Python interpreter "C:\Users\ilyao\miniforge3\envs\pysr_mamba\python.exe",
run the following code in the Python REPL:

    >>> import julia
    >>> julia.install()

For more information, see:

    https://pyjulia.readthedocs.io/en/latest/troubleshooting.html

After following the suggestion I can load the julia object but still can't import the modules directly

>>> julia.install()
[ Info: Julia executable: C:\Users\ilyao\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\bin\julia.exe
[ Info: Trying to import PyCall...
┌ Error: `import PyCall` failed
...  # The JULIA_PROJECT env variable is not defined this time
[ Info: Installing PyCall...
   Resolving package versions...
    Updating `C:\Users\ilyao\.julia\environments\v1.9\Project.toml`
  [438e738f] + PyCall v1.96.1
    Updating `C:\Users\ilyao\.julia\environments\v1.9\Manifest.toml`
  [8f4d0f93] + Conda v1.9.1
  [1914dd2f] + MacroTools v0.5.11
  [438e738f] + PyCall v1.96.1
  [81def892] + VersionParsing v1.3.0
Precompiling project...
  1 dependency successfully precompiled in 2 seconds. 71 already precompiled.

Precompiling PyCall...
Precompiling PyCall... DONE
PyCall is installed and built successfully.

>>> j = julia.core.Julia()
>>> j.eval('import PyCall; println(PyCall.libpython)')
C:\Users\ilyao\miniforge3\envs\pysr_mamba\python312.dll
>>> j.eval("Base.active_project()")
'C:\\Users\\ilyao\\.julia\\environments\\v1.9\\Project.toml'

>>> from julia import Main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Main' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\julia\__init__.py)

>>> from julia import SymbolicRegression
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'SymbolicRegression' from 'julia' (C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\julia\__init__.py)

I have not built python from scratch but the rest of the work arounds do not seem to work in the same fresh environment with pysr:

python-jl -c 'from julia.Base import banner; banner()'

Traceback (most recent call last):
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\julia\pseudo_python_cli.py", line 308, in main
    python(**vars(ns))
  File "C:\Users\ilyao\miniforge3\envs\pysr_mamba\Lib\site-packages\julia\pseudo_python_cli.py", line 52, in python
    exec(command, scope)
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'julia.Base'
>>> from julia.api import Julia
>>> jl = Julia(compiled_modules=False)
>>> from julia.Base import banner; banner()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'julia.Base'
MilesCranmer commented 11 months ago

I'm just seeing that you are using Python 3.12. Have you tried on Python 3.11? But we should probably add testing on 3.12 now that it is out.

IlyaOrson commented 11 months ago

I just realized that conda environments are not supported by PyJulia at the bottom of the troubleshooting guide.

After creating a fresh env with venv instead and installing SymbolicRegression manually I got it to work

python -m venv pysr
pysr\Scripts\activate
(pysr) python -m pip install pysr
(pysr) python -m pysr install

...
JULIA_PROJECT = @pysr-0.16.3
...
[ Info: Julia executable: C:\Users\ilyao\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\bin\julia.exe
[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed but not compatible with this Python
└ executable.  Re-building PyCall...
[ Info: Run `Pkg.build("PyCall"; verbose=true)`
...
Precompiling PyCall...
Precompiling PyCall... DONE
PyCall is installed and built successfully.
    Updating registry at `C:\Users\ilyao\.julia\registries\General`
Traceback (most recent call last):
  File "C:\Users\ilyao\miniforge3\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\ilyao\miniforge3\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\pysr\__main__.py", line 4, in <module>
    _cli(prog_name="pysr")
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\pysr\_cli\main.py", line 37, in _install
    install(julia_project, quiet, precompile)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\pysr\julia_helpers.py", line 109, in install
    _add_sr_to_julia_project(Main, io_arg)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\pysr\julia_helpers.py", line 262, in _add_sr_to_julia_project
    Main.eval("Pkg.Registry.update()")
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\julia\core.py", line 627, in eval
    ans = self._call(src)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\julia\core.py", line 555, in _call
    self.check_exception(src)
  File "D:\ilyao\Documents\research\coding\physics_informed_ADoK\pysr\lib\site-packages\julia\core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'could not load library "libgit2"
The specified module could not be found. ' occurred while calling julia code:
Pkg.Registry.update()
>>> import julia
>>> from julia import Main  # works!
>>> j = julia.core.Julia()
>>> j.eval('import Pkg; Pkg.add("SymbolicRegression")')
   Resolving package versions...
   Installed DynamicQuantities ── v0.6.3
   Installed SymbolicRegression ─ v0.22.4
    Updating `C:\Users\ilyao\.julia\environments\v1.9\Project.toml`
  [8254be44] + SymbolicRegression v0.22.4
...
Precompiling project...
  4 dependencies successfully precompiled in 84 seconds. 134 already precompiled.
>>> import numpy as np
>>>
>>> X = 2 * np.random.randn(100, 5)
>>> y = 2.5382 * np.cos(X[:, 3]) + X[:, 0] ** 2 - 0.5
>>> from pysr import PySRRegressor
>>>
>>> model = PySRRegressor(
...     niterations=40,  # < Increase me for better results
...     binary_operators=["+", "*"],
...     unary_operators=[
...         "cos",
...         "exp",
...         "sin",
...         "inv(x) = 1/x",
...         # ^ Custom operator (julia syntax)
...     ],
...     extra_sympy_mappings={"inv": lambda x: 1 / x},
...     # ^ Define operator for SymPy as well
...     loss="loss(prediction, target) = (prediction - target)^2",
...     # ^ Custom loss function (julia syntax)
... )
>>> model.fit(X, y)
Compiling Julia backend...
┌ Warning: You are using multithreading mode, but only one thread is available. Try starting julia with `--threads=auto`.
└ @ SymbolicRegression C:\Users\ilyao\.julia\packages\SymbolicRegression\XKtla\src\SymbolicRegression.jl:546
Started!
99.0%┣██████████████████████████████████████████████████████████████████████████████████▏┫ 594/600 [00:09<00:00, 63it/s]
Expressions evaluated per second: 1.44e+05. Head worker occupation: 5.5%
...  # works!
IlyaOrson commented 11 months ago

I just tried to use the local backend and it works as well! So I think it is a matter of avoiding conda environments and using venv instead.

MilesCranmer commented 11 months ago

So after going deeper into this, I think this it is just due to #451, so going to move to that issue there.