`PyCall` cannot find `ctypes` on python 3.10.2

[EHBaozi]

As the title says, despite the fact that ctypes is present. It also causes Julia to crash.

PyCall version 1.93.0, python version 3.10.2 (managed through a miniforge installation).

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

Full stacktrace:

julia> using PyCall
ERROR: InitError: PyError (PyImport_ImportModule

The Python package ctypes could not be imported by pyimport. Usually this means
that you did not install ctypes in the Python version being used by PyCall.

PyCall is currently configured to use the Python version at:

C:\Users\invis\scoop\apps\mambaforge\current\python.exe

and you should use whatever mechanism you usually use (apt-get, pip, conda,
etcetera) to install the Python package containing the ctypes module.

One alternative is to re-configure PyCall to use a different Python
version on your system: set ENV["PYTHON"] to the path/name of the python
executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.

Another alternative is to configure PyCall to use a Julia-specific Python
distribution via the Conda.jl package (which installs a private Anaconda
Python distribution), which has the advantage that packages can be installed
and kept up-to-date via Julia.  As explained in the PyCall documentation,
set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
To install the ctypes module, you can use `pyimport_conda("ctypes", PKG)`,
where PKG is the Anaconda package that contains the module ctypes,
or alternatively you can use the Conda package directly (via
`using Conda` followed by `Conda.add` etcetera).

) 
ImportError('DLL load failed while importing _ctypes: The specified module could not be found.')

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffb4706a8c5 -- PyObject_Call at C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\python310.dll (unknown line)
in expression starting at none:0
PyObject_Call at C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\python310.dll (unknown line)
macro expansion at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\exception.jl:95 [inlined]
#107 at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:43 [inlined]
disable_sigint at .\c.jl:458 [inlined]
__pycall! at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:42 [inlined]
_pycall! at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:29
_pycall! at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:11 [inlined]
#pycall#112 at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:80 [inlined]
pycall at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:80 [inlined]
show at C:\Users\invis\.julia\packages\PyCall\L0fLP\src\exception.jl:43
showerror at .\errorshow.jl:30 [inlined]
#showerror#813 at .\errorshow.jl:88
showerror##kw at .\errorshow.jl:87
unknown function (ip: 000000003cf2a7d3)
#showerror#815 at .\errorshow.jl:103
showerror##kw at .\errorshow.jl:102
unknown function (ip: 000000003cf27883)
show_exception_stack at .\errorshow.jl:866
display_error at .\client.jl:104
jfptr_display_error_48751.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757
#invokelatest#2 at .\essentials.jl:716 [inlined]
invokelatest at .\essentials.jl:714 [inlined]
print_response at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:286
#45 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:275
jfptr_YY.45_49620.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)
with_repl_linfo at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:508
print_response at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:273
do_respond at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:844
jfptr_do_respond_49916.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757
#invokelatest#2 at .\essentials.jl:716 [inlined]
invokelatest at .\essentials.jl:714 [inlined]
run_interface at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2493
jfptr_run_interface_50086.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)
run_frontend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:1230
#49 at .\task.jl:423
jfptr_YY.49_49929.clone_1 at C:\Users\invis\scoop\apps\julia\current\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:877
Allocations: 13768873 (Pool: 13762844; Big: 6029); GC: 16

[stevengj]

Probably you have some path problem. What is PyCall.python, are you letting PyCall install its own Python distro via Conda or are you using some other Python distro?

[EHBaozi]

I'm using my own python installation (using miniconda). THe path is properly set, i.e. I have the following before using PyCall:

julia> println.(split(ENV["PATH"], ";"));
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Library\mingw-w64\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Library\usr\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Library\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\Scripts
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\bin
C:\Users\invis\scoop\apps\mambaforge\4.10.1-4\condabin
...

and PyCall is also built with this python installation.

I cannot get PyCall.python since using PyCall crashes Julia.

[tkf]

You can look at the file C:\Users\invis.julia\packages\PyCall\L0fLP\deps\deps.jl (the L0fLP part may change if you use different PyCall version). You can also try typing ]build --verbose PyCall in the REPL.

I cannot get PyCall.python since using PyCall crashes Julia.

@stevengj This is why I think #945 should have a separated config package.

[EHBaozi]

deps.jl:

const python = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4\\python.exe"
const libpython = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4\\python310.dll"
const pyprogramname = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4\\python.exe"
const pyversion_build = v"3.10.2"
const PYTHONHOME = "C:\\Users\\invis\\scoop\\apps\\mambaforge\\4.10.1-4"

"True if we are using the Python distribution in the Conda package."
const conda = false

The paths are set up ok.

---

I did a bit more digging and it turns out that this problem doesn't happen with python 3.9.4. I switched to another conda environment w/ python 3.9.4, rebuilt PyCall with it and now it's working just fine. So something in python 3.10 is making PyCall unhappy. Don't know what exactly though.

[metab0t]

I met the same problem. I use Python package from conda-forge and PyCall.jl 1.93.0 on Windows 10.

• Python 3.9.7 works fine
• Python 3.9.10 errors that it fails to import _ctypes but I can import it in Python

It seems to be some issues on conda-forge side. See https://github.com/conda-forge/python-feedstock/issues/539 https://github.com/pre-commit/pre-commit/issues/1329 The same error ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

Currently, I pin Python package at 3.9.7 as a temporary solution. https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning

[isuruf]

Can you send the logs with CONDA_DLL_SEARCH_MODIFICATION_DEBUG=1 env variable set?

[metab0t]

@isuruf When Python 3.9.10 errors, I get following log

CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Scripts - ExePrefix)

CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Library\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Library\usr\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath() :: AddDllDirectory(C:\Users\yangy\AppData\Local\Programs\Julia-1.7.1\bin\Library\mingw-w64\bin - ExePrefix)

I think it should search miniforge3 prefix rather than Julia directory.

I find your PR https://github.com/conda-forge/python-feedstock/pull/546. Thanks for quick response and patch!

[isuruf]

Should be fixed now. Update your python and it should work.

[EHBaozi]

Can confirm. Thanks!