Problem loading RAPIDS.jl

[ablaom]

Looks like a python package management issue, which leaves me a bit lost. Is this, perhaps, because I don't have a GPU?

julia> using RAPIDS
[ Info: Precompiling RAPIDS [2764e59e-7dd7-4b2d-a28d-ce06411bac13]
    CondaPkg Found dependencies: /Users/anthony/.julia/packages/PythonCall/XgP8G/CondaPkg.toml
    CondaPkg Found dependencies: /Users/anthony/Julia/RAPIDS/CondaPkg.toml
    CondaPkg Resolving changes
             + cudatoolkit
             + python
             + rapids
    CondaPkg Installing packages
error    libmamba Could not solve for environment specs
critical libmamba UnsatisfiableError
ERROR: LoadError: InitError: failed process: Process(`/Users/anthony/.julia/artifacts/bfd9e864c155203704978fba74583859175640d2/bin/micromamba -r /Users/anthony/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root create -q -y -p /var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/jl_FSq8U5/.CondaPkg/env --override-channels --no-channel-priority "cudatoolkit[version='11.5']" "python[version='3.9']" "python[version='>=3.5,<4']" "rapids[version='22.02']" -c conda-forge -c nvidia -c rapidsai`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:525 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:440
  [3] run(::Cmd)
    @ Base ./process.jl:438
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/Olo5Y/src/resolve.jl:283
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/Olo5Y/src/resolve.jl:191
  [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg ~/.julia/packages/CondaPkg/Olo5Y/src/resolve.jl:395
  [7] resolve()
    @ CondaPkg ~/.julia/packages/CondaPkg/Olo5Y/src/resolve.jl:288
  [8] envdir()
    @ CondaPkg ~/.julia/packages/CondaPkg/Olo5Y/src/env.jl:70
  [9] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/context.jl:56
 [10] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/CPython.jl:21
 [11] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:696
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:782
 [13] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1020
 [14] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:936
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:923
 [16] include
    @ ./Base.jl:384 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1235
 [18] top-level scope
    @ none:1
 [19] eval
    @ ./boot.jl:360 [inlined]
 [20] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [21] top-level scope
    @ none:1
during initialization of module C
in expression starting at /Users/anthony/Julia/RAPIDS/src/RAPIDS.jl:1
ERROR: Failed to precompile RAPIDS [2764e59e-7dd7-4b2d-a28d-ce06411bac13] to /Users/anthony/.julia/compiled/v1.6/RAPIDS/jl_sLzIBl.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1385
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1329
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1043
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:936
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:923

(jl_FSq8U5) pkg> st
      Status `/private/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/jl_FSq8U5/Project.toml`
  [add582a8] MLJ v0.18.2
  [2764e59e] RAPIDS v0.1.0 `~/Julia/RAPIDS`

julia> versioninfo()
Julia Version 1.6.5
Commit 9058264a69 (2021-12-19 12:30 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.6.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_LTS_PATH = /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia
  JULIA_PATH = /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia
  JULIA_EGLOT_PATH = /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia
  JULIA_NUM_THREADS = 5
  JULIA_NIGHTLY_PATH = /Applications/Julia-1.7.app/Contents/Resources/julia/bin/julia

[ablaom]

Also, I have PYTHON=/Users/anthony/anaconda2/envs/py37/bin/python which is python 3.7. Maybe that's it. I'll try updating my python installation and adding those libraries manually. And then get back to you.

  CondaPkg Resolving changes
             + cudatoolkit
             + python
             + rapids

[tylerjthomas9]

Looks like a python package management issue, which leaves me a bit lost. Is this, perhaps, because I don't have a GPU?

Unfortunately, RAPIDS requires linux, and a Pascal or more recent NVIDIA gpu with at least 6 TFLOPS (1070 or better). I hope that platform support will expand over time.

Also, I have PYTHON=/Users/anthony/anaconda2/envs/py37/bin/python which is python 3.7. Maybe that's it. I'll try updating my python installation and adding those libraries manually. And then get back to you.

I used CondaPkg.jl, which has micromamba instead of conda, and installs a fresh environment in the project directory instead of utilizing other environments. This was for two reasons, the first is that the python environment will hopefully be fully contained to avoid errors when installing the RAPIDS python libraries. This will hopefully make it easier for other developers to use the library. I have personally had a lot of headaches with RAPIDS compatibility and breaking python environments. The second is that micromamba gives us an order of magnitude quicker installation of all the RAPIDS dependencies.

However, I am not married to this approach. I am open to suggestions/changes to any part of the library.

I have tried it on Google Colab, but getting a compatible GPU seems less and less common as time goes on. Additionally, the factory reset option seems to have been replaced with a disconnect and delete runtime, which then gives you a captcha. Here is a Google Colab link for testing the library (if you get lucky with gpu allocation): https://colab.research.google.com/drive/11Cpsm4vwLJjYcbhjdXJtBrHEnb54hIsL?usp=sharing

[ablaom]

Thanks for those helpful and exhaustive comments. I will wait for my University to re-activate my GPU allocation and go from there.

By the way the MLJ interface looks very good - but I'd like to play around with this some more before compiling my feedback. Thanks for your continued patience.

[ablaom]

If I change PYTHON="" then my issue is resolved. Closing, as I think this confirms this is a python package mngmt issue over which RAPIDS.jl has no control.