Closed thealanjason closed 4 months ago
In the default (@v1.10)
environment I get:
using CondaPkg
] conda run python
Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 07:53:56) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PATH']
'C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Library\\mingw-w64\\bin;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Library\\usr\\bin;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Library\\bin;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Scripts;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\bin;........
This shows that the correct paths have been added in the default Julia enironment.
I modified the init()
function here locally to print os.environ['PATH']
and get the following output:
D:\psimpy.jl> julia --project=.
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.0 (2023-12-25)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PythonCall
CondaPkg Found dependencies: D:\psimpy.jl\CondaPkg.toml
CondaPkg Found dependencies: C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\CondaPkg.toml
CondaPkg Found dependencies: C:\Users\Alan Jason Correa\.julia\environments\v1.10\CondaPkg.toml
CondaPkg Dependencies already up to date
D:\psimpy.jl\.CondaPkg\env;D:\psimpy.jl\.CondaPkg\env\Library\mingw-w64\bin;D:\psimpy.jl\.CondaPkg\env\Library\usr\bin;D:\psimpy.jl\.CondaPkg\env\Library\bin;D:\psimpy.jl\.CondaPkg\env\Scripts;D:\psimpy.jl\.CondaPkg\env\bin;......
Quoting from the OP:
Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library.
The os.environ['PATH']
is ignored on Windows. So I tried to debug by doing something very hacky: Made locally some changes to the init()
function in __init__.py
file of pysrc/juliacall
module as follows:
def init():
import os
#D:\psimpy.jl\.CondaPkg\env;
path = os.path.join(os.getcwd(), ".CondaPkg", "env")
print(f"Adding {path} to dll search path")
os.add_dll_directory(path)
#D:\psimpy.jl\.CondaPkg\env\Library\mingw-w64\bin;
path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Library", "mingw-w64", "bin")
print(f"Adding {path} to dll search path")
os.add_dll_directory(path)
# # D:\psimpy.jl\.CondaPkg\env\Library\usr\bin; ##### ERROR: Directory not found.
# path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Library", "usr", "bin")
# print(f"Adding {path} to dll search path")
# os.add_dll_directory(path)
# D:\psimpy.jl\.CondaPkg\env\Library\bin;
path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Library", "bin")
print(f"Adding {path} to dll search path")
os.add_dll_directory(path)
# D:\psimpy.jl\.CondaPkg\env\Scripts;
path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Scripts")
print(f"Adding {path} to dll search path")
os.add_dll_directory(path)
# D:\psimpy.jl\.CondaPkg\env\bin
path = os.path.join(os.getcwd(), ".CondaPkg", "env", "bin")
print(f"Adding {path} to dll search path")
os.add_dll_directory(path)
And this fixes the issue. See below:
julia>
PS D:\psimpy.jl> julia --project=.
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.0 (2023-12-25)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PythonCall
Adding D:\psimpy.jl\.CondaPkg\env to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\Library\mingw-w64\bin to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\Library\bin to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\Scripts to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\bin to dll search path
julia> np = pyimport("numpy")
Python: <module 'numpy' from 'D:\\psimpy.jl\\.CondaPkg\\env\\lib\\site-packages\\numpy\\__init__.py'>
Maybe there is a better way or location to make the os.add_dll_directory(<dll_search_path>)
calls. However, I don't know how the PythonCall package is structured and its inner workings. It seems very difficult for me to understand. I could create a pull request for now.
What you are experiencing is actually a bug in Python 3.10.0 which has been fixed in 3.10.2.
From this line of CondaPkg.status()
python v3.10.0 (3.10)
I can see that you have the buggy Python 3.10.0 installed. This happened because you added Python like CondaPkg.add("python", version="3.10"). Unlike Julia's package manager, a version of
3.10in Conda means exactly
3.10.0`.
You probably actually wanted version="3.10.*"
to get the latest Python 3.10.
Note that this prevents any newer Python from being used. Unless you really do need Python 3.10 it's generally recommended to do something like version=">=3.10,<4"
to get any Python 3.10+.
Closing as I believe this issue is solved.
Affects: PythonCall / JuliaCall(Maybe)
Describe the bug Using a project folder environment for a Julia project with
julia --project=.
and when trying to use PythonCall along with CondaPkg with numpy on Windows 10, with Julia 1.10 I get the following error:This however does not happen when running in the default Julia environment
(@v1.10)
Expectation:
using PythonCall
should work without issues (as on Linux)Steps to Reproduce
julia --project=.
inside a temporary folder OR start Julia and] activate .
Your system OS: Windows10 Julia Version: 1.10 Python Version: 3.10 PythonCall Version: v0.9.15 Output of Pkg.status():
Output of CondaPkg.status():
Additional context The issues #146, #245, #339, #373 seem to also be affected by this.
On a bit of further reading on stack overflow, I found out that this issue is related to loading the DLLs on Windows. The Python 3.8 release notes state:
DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)