python.exe not found - Githubissues

gyansinha commented 11 months ago

IOError: could not spawn 'H:\.julia\environments\v1.9\.CondaPkg\env\python.exe' --version: no such file or directory (ENOENT)

It looks like the python executable isn't there at all after "add PythonCall" - any ideas why this may be happening.

cjdoris commented 11 months ago

It'll be much easier to help if you give the full sequence of steps you did to get to that error, plus the full error message.

cjdoris commented 11 months ago

My guess though is that your antivirus deleted python.exe after it was installed.

gyansinha commented 11 months ago

It'll be much easier to help if you give the full sequence of steps you did to get to that error, plus the full error message.

add PythonCall using PythonCall add quantlib

] conda resolve

This is the error I get:

CondaPkg Creating environment
         │ H:\.julia\artifacts\4b87dc8b933ca2d289a392e264aaf44df2410f6f\bin\micromamba.exe
         │ -r H:\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root
         │ create
         │ -y
         │ -p \\PROD-FILE\XXXX$\User\gsinha\.julia\dev\Clover\.CondaPkg\env
         │ --override-channels
         │ --no-channel-priority
         │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
         │ quantlib[version='*']
         └ -c conda-forge

critical libmamba remove: Access is denied.: "\PROD-FILE\XXXX$\User\gsinha.julia\dev\Clover.CondaPkg\env\DLLs_ctypes.pyd.mamba_trash" ERROR: failed process: Process('H:\.julia\artifacts\4b87dc8b933ca2d289a392e264aaf44df2410f6f\bin\micromamba.exe' -r 'H:\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p '\\PROD-FILE\XXXX$\User\gsinha\.julia\dev\Clover\.CondaPkg\env' --override-channels --no-channel-priority "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" "quantlib[version='*']" -c conda-forge, ProcessExited(1)) [1]

Stacktrace: [1] pipeline_error @ .\process.jl:565 [inlined] [2] run(::Cmd; wait::Bool) @ Base .\process.jl:480 [3] run(::Cmd) @ Base .\process.jl:477 [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any) @ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:398 [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any) @ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:299 [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool) @ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:547 [7] resolve(; force::Bool) @ CondaPkg.PkgREPL H:.julia\packages\CondaPkg\22jbl\src\PkgREPL.jl:82 [8] resolve() @ CondaPkg.PkgREPL H:.julia\packages\CondaPkg\22jbl\src\PkgREPL.jl:81 [9] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL) @ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:409 [10] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool) @ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:390 [11] do_cmd @ C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:380 [inlined] [12] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool) @ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:557 [13] #invokelatest#2 @ .\essentials.jl:816 [inlined] [14] invokelatest @ .\essentials.jl:813 [inlined] [15] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState) @ REPL.LineEdit C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\REPL\src\LineEdit.jl:2647 [16] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef) @ REPL C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\REPL\src\REPL.jl:1300 [17] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})() @ REPL .\task.jl:514

I am having the anti-virus looked into - thanks for your tip on that.

gyansinha commented 11 months ago

After this step, I do the followiing after exiting Julia entirely:

julia> using PythonCall ERROR: InitError: Python executable "h:\.julia\dev\Clover\.CondaPkg\env\python.exe" is not executable. Stacktrace: [1] error(s::String) @ Base .\error.jl:35 [2] init_context() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\context.jl:88 [3] init() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\CPython.jl:21 [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String) @ Base .\loading.jl:1115 [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}) @ Base .\loading.jl:1061 [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128) @ Base .\loading.jl:1506 [7] _require(pkg::Base.PkgId, env::String) @ Base .\loading.jl:1783 [8] _require_prelocked(uuidkey::Base.PkgId, env::String) @ Base .\loading.jl:1660 [9] macro expansion @ .\loading.jl:1648 [inlined] [10] macro expansion @ .\lock.jl:267 [inlined] [11] require(into::Module, mod::Symbol) @ Base .\loading.jl:1611 during initialization of module C

caused by: IOError: could not spawn 'h:\.julia\dev\Clover\.CondaPkg\env\python.exe' --version: no such file or directory (ENOENT) Stacktrace: [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) @ Base .\process.jl:128 [2] #760 @ .\process.jl:139 [inlined] [3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) @ Base .\process.jl:223 [4] _spawn @ .\process.jl:138 [inlined] [5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) (repeats 2 times) @ Base .\process.jl:166 [6] run(::Base.CmdRedirect; wait::Bool) @ Base .\process.jl:479 [7] run @ .\process.jl:477 [inlined] [8] init_context() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\context.jl:86 [9] init() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\CPython.jl:21 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String) @ Base .\loading.jl:1115 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}) @ Base .\loading.jl:1061 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128) @ Base .\loading.jl:1506 [13] _require(pkg::Base.PkgId, env::String) @ Base .\loading.jl:1783 [14] _require_prelocked(uuidkey::Base.PkgId, env::String) @ Base .\loading.jl:1660 [15] macro expansion @ .\loading.jl:1648 [inlined] [16] macro expansion @ .\lock.jl:267 [inlined] [17] require(into::Module, mod::Symbol) @ Base .\loading.jl:1611

cjdoris commented 11 months ago

Can you remove that .CondaPkg folder entirely and try again?

schlichtanders commented 11 months ago

I am running into a similar behaviour by first installing some packages with CondaPkg, before even installing PythonCall.

It seems to me that this order is not supported currently. A workaround is to first install PythonCall into the environment before the first time import CondaPkg is executed in that environment. Then the __init__ of CondaPkg will automatically pickup all dependencies from PythonCall.

Unfortunately that workaround does not work for me, because my package depends on RCall, which I like to build on top of CondaPkg. However unlike PythonCall, RCall requires a ready R installation before it can be installed, hence I need to import CondaPkg before installing my package which depends on R. Hence all the Conda dependencies from my package are neither picked up nor resolved by CondaPkg.

So for me the only workaround left is to

delete .CondaPkg after RCall is installed, using rm(dirname(CondaPkg.envdir()), recursive=true)
and run import CondaPkg; CondaPkg.resolve() again after I added my package.

cjdoris commented 10 months ago

@schlichtanders That order should definitely work.

Can you give the sequence of steps to reproduce your error? And also give the full error you see.

schlichtanders commented 10 months ago

It was similar to

julia -e 'import Pkg; \
    Pkg.Registry.add("General"); \
    Pkg.add("CondaPkg"); \
    import CondaPkg; \
    CondaPkg.add("r-base"); \
    ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"; \
    Pkg.add("RCall"); \
    Pkg.add("PythonCall"); \
    Pkg.precompile(); \
'

schlichtanders commented 10 months ago

Here a complete reproducible example

Start a docker container with docker run -it --rm julia:1.9-bookworm bash

run the above installation instructions

julia -e 'import Pkg; \
Pkg.Registry.add("General"); \
Pkg.add("CondaPkg"); \
import CondaPkg; \
CondaPkg.add("r-base"); \
ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"; \
Pkg.add("RCall"); \
Pkg.add("PythonCall"); \
Pkg.precompile(); \
'

run julia -e "import PythonCall"

then the output is

ERROR: InitError: Python executable "/root/.julia/environments/v1.9/.CondaPkg/env/bin/python" is not executable.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:88
  [3] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
  [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
  [7] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
  [8] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
  [9] macro expansion
    @ ./loading.jl:1648 [inlined]
 [10] macro expansion
    @ ./lock.jl:267 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
during initialization of module C

caused by: IOError: could not spawn `/root/.julia/environments/v1.9/.CondaPkg/env/bin/python --version`: no such file or directory (ENOENT)
Stacktrace:
  [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:128
  [2] #760
    @ ./process.jl:139 [inlined]
  [3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:223
  [4] _spawn
    @ ./process.jl:138 [inlined]
  [5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, IO}}) (repeats 2 times)
    @ Base ./process.jl:166
  [6] run(::Base.CmdRedirect; wait::Bool)
    @ Base ./process.jl:479
  [7] run
    @ ./process.jl:477 [inlined]
  [8] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:86
  [9] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
 [14] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [15] macro expansion
    @ ./loading.jl:1648 [inlined]
 [16] macro expansion
    @ ./lock.jl:267 [inlined]
 [17] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611

Note that this is only a dummy example. Instead of PythonCall the package which is actually needed is MyPackage.

MyPackage depends on RCall AND CondaPkg, defining a couple of R dependencies via CondaPkg.toml.
Hence MyPackage is a bit like PythonCall in this example, in that it depends on CondaPkg.
And a bit like RCall in that adding/importing MyPackage would build RCall itself, but RCall's R_HOME should be set by CondaPkg.envdir(). Hence MyPackage needs to be added after CondaPkg.resolve()/CondaPkg.envdir().

The ideal solution for me would be that RCall is resolving its dependency dynamically, but as of now it is not foreseeable if this will ever happen.

It would be great if CondaPkg can resolve newly added other julia dependencies which also include CondaPkg dependencies.

cjdoris commented 10 months ago

Ok I see, so the issue is that you (implicitly) resolve the Conda environment in order to build RCall and then load some other packages, which also have Conda dependencies, and these aren't installed because CondaPkg doesn't auto resolve this time.

The reason for this is that CondaPkg doesn't resolve if you've already resolved, unless you explicitly add any packages.

A nice solution that I think you were getting at is to also resolve if the installed Julia packages change.

Is it possible to cheaply detect if the Julia environment has changed? Maybe Pkg tracks the time that the last resolve happened? Or we could see if Pkg.project() changed but that might be quite a slow check.

cjdoris commented 10 months ago

Ok after a discussion on Slack the best approach seems to be to check if the mtime/hash of Manifest.toml changed since the last resolve.

schlichtanders commented 10 months ago

Sounds great! Thank you for inspecting/discussing this further

JuliaPy / PythonCall.jl

python.exe not found #381