search

MilesCranmer / PySR

High-Performance Symbolic Regression in Python and Julia
https://astroautomata.com/PySR
Apache License 2.0
2.32k stars 211 forks source link

Windows Julia Install - could not load library "libpcre2-8" The specified module could not be found. #566

Closed tbuckworth closed 6 months ago

tbuckworth commented 6 months ago

What happened?

After installing PySR on windows, on the first import of the module, the Julia install starts, but fails with this error message:

... fatal: error thrown and no exception handler available. InitError(mod=:Sys, error=ErrorException("could not load library "libpcre2-8" The specified module could not be found. ")) ijl_errorf at C:/workdir/src\rtutils.c:77 ...

I located libpcre2-8 in the virtual environment folder: ...\venv\julia_env\pyjuliapkg\install\bin\libpcre2-8.dll

I found this issue on the julia repository, but it has no solution given.

Does anyone know of a workaround?

Would installing julia separately (outside of venv) help?

Version

0.17.2

Operating System

Windows

Package Manager

pip

Interface

Script (i.e., python my_script.py)

Relevant log output

No response

Extra Info

No response

MilesCranmer commented 6 months ago

Looks like there is more info here: https://github.com/JuliaLang/julia/issues/52007

Can you try to install Julia manually with different methods? See here: https://julialang.org/downloads/

(You should try to install a version of Julia other than 1.10.1 or 1.10.2. Other versions should be good)

If you can get Julia working itself, then juliacall should be able to connect to it.

tbuckworth commented 6 months ago

Thanks for your swift response.

I have now downloaded Julia 1.14.5.0 for Windows.

I can use it independently and it works.

Running my script again didn't make any difference, same error thrown.

I deleted julia_env folder in my venv and have re-run the script (just importing pysr) - still same error thrown.

I will see if uninstalling and re-installing pysr still has the same problem.

tbuckworth commented 6 months ago

I tried uninstalling juliacall, juliapkg and pysr, then installed pysr and ran again.

Still have the same error thrown.

MilesCranmer commented 6 months ago

Julia 1.14.5.0 does not exist (1.10.2 is the max version). Could you check the version of Julia you are installing?

tbuckworth commented 6 months ago

Sorry for the confusion.

Strangely, Windows said 1.14.5, but the directory of .julia said 1.10.X, which I have now deleted (not certain which X it was unfortunately).

I installed 1.6.7 and have run my script again, but the output said that it is installing and using 1.10.0, (I guess 1.6.7 is too old?)

It then threw the same error.

On the download page, I can only find the options to install 1.10.2, 1.6.7 or the 1.11.0 version.

I'm not really sure what to try now.

MilesCranmer commented 6 months ago

I installed 1.6.7 and have run my script again, but the output said that it is installing and using 1.10.0, (I guess 1.6.7 is too old?)

It seems like it's simply not finding the Julia you installed, otherwise it wouldn't try to install Julia again. Where did you install it to?

MilesCranmer commented 6 months ago

Also see the options here: https://juliapy.github.io/PythonCall.jl/stable/juliacall/#julia-config

You can specify the Julia binary location with the environment variable PYTHON_JULIACALL_BINDIR. Or you can simply call Python with python -X juliacall-home=/path/to/julia/folder.

tbuckworth commented 6 months ago

I've tried two things:

  1. Added this to my script: os.environ["PYTHON_JULIACALL_BINDIR"] = r"C:\Users\titus\PycharmProjects\train-procgen-pytorch\venv\julia_env\pyjuliapkg\install\bin"

When I run the script, this output appears:

[juliapkg] Using Julia 1.10.0 at C:\Users\titus\PycharmProjects\train-procgen-pytorch\venv\julia_env\pyjuliapkg\install\bin\julia.exe

But in the end the same error is thrown.

  1. Added this to my script: os.environ["PYTHON_JULIACALL_BINDIR"] = r"C:\Users\titus\AppData\Local\Microsoft\WindowsApps" Which is where I have installed Julia manually. Then I deleted the ..\venv\julia_env\ folder, uninstalled juliacall, juliapkg, pysr, reinstalled pysr.

When I run the script, this output appears:

[juliapkg] Installing Julia 1.10.0 using JuliaUp [juliapkg] WARNING: You have Julia 1.10.2 installed but ~1.6.1, ~1.7, ~1.8, ~1.9, =1.10.0, ^1.10.3 is required. [juliapkg] It is recommended that you upgrade Julia or install JuliaUp. [juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json [juliapkg] WARNING: About to install Julia 1.10.0 to C:\Users\titus\PycharmProjects\train-procgen-pytorch\venv\julia_env\pyjuliapkg\install. [juliapkg] If you use juliapkg in more than one environment, you are likely to have Julia [juliapkg] installed in multiple locations. It is recommended to install JuliaUp [juliapkg] (https://github.com/JuliaLang/juliaup) or Julia (https://julialang.org/downloads) [juliapkg] yourself. [juliapkg] Downloading Julia from https://julialang-s3.julialang.org/bin/winnt/x64/1.10/julia-1.10.0-win64.zip

And then the same error results.

But I don't believe I do have julia 1.10.2 installed.

I am currently going through my pc trying to make sure I uninstall/delete all Julia installs/folders to start again

MilesCranmer commented 6 months ago

What is the path to the actual Julia exe application? Not the directory but the full path? (And can you verify you can run Julia normally?)

if you start julia can you paste the output of versioninfo()? It should show a version other than 1.10.2 or 1.10.1 if installed correctly.

tbuckworth commented 6 months ago

Julia is here (i get this when i run where julia):

C:\Users\titus\AppData\Local\Microsoft\WindowsApps\julia.exe

But there is also a location here:

C:\Users\titus.julia\juliaup\julia-1.10.2+0.x64.w64.mingw32\bin\julia.exe

and here:

C:\Users\titus.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\bin\julia.exe

If I run the first one and then versioninfo(), I get

Julia Version 1.10.0 Commit 3120989f39 (2023-12-25 18:01 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: 8 × Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-15.0.7 (ORCJIT, icelake-client) Threads: 1 on 8 virtual cores

On the second version i get this:

Julia Version 1.10.2 Commit bd47eca2c8 (2024-03-01 10:14 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: 8 × Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-15.0.7 (ORCJIT, icelake-client) Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)

the third gives me:

Julia Version 1.10.0 Commit 3120989f39 (2023-12-25 18:01 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: 8 × Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-15.0.7 (ORCJIT, icelake-client) Threads: 1 on 8 virtual cores

If i run juliaup remove 1.10.2 i get:

Error: '1.10.2' cannot be removed because it is currently not installed.

if i run juliaup default 1.10.0 i get:

Configured the default Julia version to be '1.10.0'.

If i delete C:\Users\titus\.julia\juliaup\julia-1.10.2+0.x64.w64.mingw32 folder and then uninstall and reinstall pysr, I get the same error, even if i use this os.environ["PYTHON_JULIACALL_BINDIR"] = r"C:\Users\titus\AppData\Local\Microsoft\WindowsApps" or this: os.environ["PYTHON_JULIACALL_BINDIR"] = r"C:\Users\titus\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\bin"

Even in my final attempt the output was:

[juliapkg] Locating Julia ~1.6.1, ~1.7, ~1.8, ~1.9, =1.10.0, ^1.10.3 [juliapkg] Using Julia 1.10.0 at C:\Users\titus.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\bin\julia.exe [juliapkg] Using Julia project at C:\Users\titus\PycharmProjects\train-procgen-pytorch\venv\julia_env [juliapkg] Installing packages:

Indicating that it was indeed using 1.10.0, but in the end it still had the same error:

fatal: error thrown and no exception handler available. InitError(mod=:Sys, error=ErrorException("could not load library "libpcre2-8" The specified module could not be found. ")) ijl_errorf at C:/workdir/src\rtutils.c:77 ijl_load_dynamic_library at C:/workdir/src\dlload.c:388 jl_getlibrary at C:/workdir/src\runtime_ccall.cpp:46 jl_getlibrary at C:/workdir/src\runtime_ccall.cpp:38 [inlined] ijl_load_and_lookup at C:/workdir/src\runtime_ccall.cpp:59 jlplt_pcre2_compile_8_75038.1 at C:\Users\titus.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)

I've resorted to using WSL, but I'd still like to be able to use windows.

Thanks for your help so far

MilesCranmer commented 6 months ago

It's good that Julia actually starts when you run it normally. Make sure to use the 1.10.0 ones and delete the 1.10.2 one.

Now the remaining issue is why PythonCall.jl cannot actually open that one. For this I'm a bit confused why it's not working on your system but works in other Windows machines (like the GitHub action). Have you looked at the discussion on https://github.com/JuliaLang/julia/issues/52007 which is specific to UWP?

This issue seems wider than just PySR so you might also try to post about this issue on the JuliaLang discourse. People are very helpful there and have wide expertise: https://discourse.julialang.org/

tbuckworth commented 6 months ago

I have read the #52007 thread, but not found a solution there. I'll look into posting on the JuliaLang discourse. Thanks for your help

MilesCranmer commented 6 months ago

Okay, let me know how it goes. Would be useful to link your discourse thread here for posterity.

xiongxin9000 commented 5 months ago

Hi Miles

I am facing the same issue,

can you please provide more detailed solution.

Thank you.

MilesCranmer commented 5 months ago

@tbuckworth do you think you could share if you were able to solve it and how you did?

@xiongxin9000 can you paste detailed system diagnostics and everything you have tried thus far? And also verify you can run normal Julia programs, and also run from juliacall import jl within Python. (i.e., no use of PySR itself yet)

xiongxin9000 commented 5 months ago

Hi Miles,

Thank you for response. I tried reinstalling Julia version 1.9.4 according to this post's answer before. But the error is still there. I can type Julia into Julia's prompt. I tried with from juliacall import jl within Python. It gives me the same error without PySR.

MilesCranmer commented 5 months ago

Okay so this sounds like at the level of juliacall rather than PySR. Which is fine, we can debug it here, but good to know.

Can you confirm you can run commands within Julia itself? Like if you just start up Julia outside of Python and run stuff.

It might also be useful to see if you can run a regexp within Julia itself as this is related to the missing library.

@cjdoris seems like some users run into this, not sure you know what it’s from? Is it maybe the type of binary being installed?

xiongxin9000 commented 5 months ago

I tried to run Julia in windows terminal. simple hello world it works,

julia> "Hello World"
"Hello World"

julia> myregex = r"^\s*(?:#|$)"
r"^\s*(?:#|$)"

julia> typeof(myregex)
Regex

And also the regexp

Thank you I

tbuckworth commented 5 months ago

@tbuckworth do you think you could share if you were able to solve it and how you did?

@xiongxin9000 can you paste detailed system diagnostics and everything you have tried thus far? And also verify you can run normal Julia programs, and also run from juliacall import jl within Python. (i.e., no use of PySR itself yet)

@MilesCranmer - I ended up installing Ubuntu instead. I didn't get around to posting on the JuliaLang discourse, unfortunately.

JanisErdmanis commented 4 months ago

Julia is here (i get this when i run where julia):

C:\Users\titus\AppData\Local\Microsoft\WindowsApps\julia.exe

When applications are launched from WindowsApps folder they by default are sandboxed. It is possible to add exemption in AppxManifest.xml granting full trust privileges for the installation to circumvent the sandboxing where the libpcre error does not occur. Meanwhile someone needs to figure out how to solve the underlying issue with Julia.

I am curios though how did the Julia installation ended up into WindowsApps folder! Is it installed by pip there?

MilesCranmer commented 3 months ago

Seems like @Angelld23 found a workaround on https://github.com/JuliaLang/julia/issues/52007#issuecomment-2173945505. Copying here for visibility:

import os import ctypes import glob

# Path to the bin directory of your Julia installation
julia_bin_path = "C:\path\to\your\dll_directory" # (which is the same as the julia.exe)

# Add the bin directory to PATH
os.environ["PATH"] += ";" + julia_bin_path

# Load each DLL file in the bin directory
for dll_path in glob.glob(os.path.join(julia_bin_path, "*.dll")):
    try:
        ctypes.CDLL(dll_path)
        print(f"Loaded {dll_path} successfully.")
    except OSError as e:
        print(f"Could not load {dll_path}: {e}")

from pysr import PySRRegressor