Import error caused by difference between `Reference from:` and `Expected in:`

[facusapienza21]

Hi,

I am using the last version of CondaPkg=0.2.26 that now allows to set openssl_version: "ignore", which based on the solution to this other issue I posted in PythonCall should solve the problem.

Now, I am getting the following error when trying to import Sleipnir.jl using CondaPkg + PythonCall. My Python environment requires rioxarray, which needs libxml2 and libspatialite as dependencies. For some reason, it seems that CondaPkg is trying to import these from .julia/artifacts. I don't know if this is a desired behaviour, or it is something I can change.

This issue is a follow up on the post in Julia Discourse. Sorry for the insistence, but this is fully blocking the development of our library and I imagine will be a common issue for other people working in spatial-related sciences.

[cjdoris]

My first suggestion would be to see if it is affected by loading order. Try doing using OpenSSL_jll before anything else. Then in a fresh session try using PythonCall; pyimport("openssl") before anything else. Does anything change?

[cjdoris]

Secondly, it's hard to help without being able to reproduce your problem. Please post code to replicate your issue from scratch, and post the complete output you get too. Please don't use screenshots, post the actual text.

[facusapienza21]

Thank you @cjdoris for your response :)

I tried your first suggestion. For clarity, this is the workflow:

1. In a Julia session, I activate my environment, I instantiated it, and do an using Open_SSL
2. I exit the session and open a new Julia session
3. In the new julia session, I activate my environment, followed by using PythonCall; pyimport("ssl") (Notice that I am doing an import of ssl, not openssl as originally suggested)
4. I finally do a using Sleipnir

This workflow works and does not give any error message! Immediately after I do the using Sleipnir, I see the following message

Python: <module 'ssl' from '/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/python3.12/ssl.py'>

which seems to indicate that is finding the right ssl file, I guess?

But I am not sure why this works... does this suggest something to you @cjdoris ?

The code to reproduce this behaviour can be found in this version of the GitHub repository of the project. I can post the raw code here, but I imagine is going to be more difficult to read. The error should be reproducible by just instantiating and using the version of Sleipnir from the repo.

Thank you so much!

[facusapienza21]

Importantly, even when the error does not show up when doing using Sleipnir as it did before, now the same error is showing when manually running the tests:

include("test/runtests.jl")

when I get

  Activating project at `~/.julia/dev/Sleipnir`
lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.2.dylib
Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.dylib
lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.8.dylib
Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.dylib
2024-11-20 15:10:30: oggm.cfg: Reading default parameters from the OGGM `params.cfg` configuration file.
2024-11-20 15:10:30: oggm.cfg: Multiprocessing switched OFF according to the parameter file.
2024-11-20 15:10:30: oggm.cfg: Multiprocessing: using all available processors (N=11)
2024-11-20 15:10:30: oggm.utils: Checking the download verification file checksum...
2024-11-20 15:10:32: oggm.cfg: PARAMS['dl_verify'] changed from `True` to `False`.
2024-11-20 15:10:32: oggm.cfg: PARAMS['continue_on_error'] changed from `False` to `True`.
2024-11-20 15:10:32: oggm.cfg: PARAMS['border'] changed from `40` to `10`.
Test Summary:                                 | Pass  Total  Time
Parameters constructors with specified values |    4      4  3.9s
Number of cores: 4
Number of workers: 3
      From worker 4:    ┌ Info: CondaPkg: Waiting for lock to be freed. You may delete this file if no other process is resolving.
      From worker 4:    └   lock_file = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/lock"
      From worker 3:    ┌ Info: CondaPkg: Waiting for lock to be freed. You may delete this file if no other process is resolving.
      From worker 3:    └   lock_file = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/lock"
      From worker 2:        CondaPkg Found dependencies: /Users/sapienza/.julia/dev/Sleipnir/CondaPkg.toml
      From worker 2:        CondaPkg Found dependencies: /Users/sapienza/.julia/packages/PythonCall/Nr75f/CondaPkg.toml
      From worker 2:        CondaPkg Dependencies already up to date
      From worker 2:    lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
      From worker 2:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.2.dylib
      From worker 2:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.dylib
      From worker 2:    lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
      From worker 4:    lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
      From worker 2:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.8.dylib
      From worker 2:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.dylib
      From worker 4:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.2.dylib
      From worker 4:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.dylib
      From worker 4:    lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
      From worker 4:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.8.dylib
      From worker 4:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.dylib
      From worker 3:    lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
      From worker 3:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.2.dylib
      From worker 3:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libxml2.dylib
      From worker 3:    lib_dir = "/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib"
      From worker 3:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.8.dylib
      From worker 3:    Opened /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/libspatialite.dylib
Parameters constructors by default: Error During Test at /Users/sapienza/.julia/dev/Sleipnir/test/runtests.jl:20
  Got exception outside of a @test
  On worker 2:
  InitError: Python: ImportError: dlopen(/Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/python3.12/lib-dynload/_ssl.cpython-312-darwin.so, 0x0002): Symbol not found: _X509_STORE_get1_objects
    Referenced from: <B1CE67CC-8161-3609-8F45-6B9EC2F14447> /Users/sapienza/.julia/dev/Sleipnir/.CondaPkg/env/lib/python3.12/lib-dynload/_ssl.cpython-312-darwin.so
    Expected in:     <4C4C44CB-5555-3144-A104-41A9AFACD8FB> /Users/sapienza/.julia/artifacts/e33c9389fd079973c7a7520d36779612df7ec447/lib/libcrypto.3.dylib
  Python stacktrace:
   [1] <module>
     @ ssl ~/.julia/dev/Sleipnir/.CondaPkg/env/lib/python3.12/ssl.py:100
   [2] <module>
     @ oggm.utils._downloads ~/.local/lib/python3.12/site-packages/oggm/utils/_downloads.py:24
   [3] <module>
     @ oggm.utils ~/.local/lib/python3.12/site-packages/oggm/utils/__init__.py:2
   [4] <module>
     @ oggm ~/.local/lib/python3.12/site-packages/oggm/__init__.py:39
  Stacktrace:
    [1] pythrow
      @ ~/.julia/packages/PythonCall/Nr75f/src/Core/err.jl:92
    [2] errcheck
      @ ~/.julia/packages/PythonCall/Nr75f/src/Core/err.jl:10 [inlined]
    [3] pyimport
      @ ~/.julia/packages/PythonCall/Nr75f/src/Core/builtins.jl:1561
    [4] __init__
      @ ~/.julia/dev/Sleipnir/src/setup/config.jl:27
    [5] run_module_init
      @ ./loading.jl:1134
    [6] register_restored_modules
      @ ./loading.jl:1122
    [7] _include_from_serialized
      @ ./loading.jl:1067
    [8] _require_search_from_serialized
      @ ./loading.jl:1581
    [9] _require
      @ ./loading.jl:1938
   [10] __require_prelocked
      @ ./loading.jl:1812
   [11] #invoke_in_world#3
      @ ./essentials.jl:926 [inlined]
   [12] invoke_in_world
      @ ./essentials.jl:923 [inlined]
   [13] _require_prelocked
      @ ./loading.jl:1803
   [14] macro expansion
      @ ./loading.jl:1790 [inlined]
   [15] macro expansion
      @ ./lock.jl:267 [inlined]
   [16] __require
      @ ./loading.jl:1753
   [17] #invoke_in_world#3
      @ ./essentials.jl:926 [inlined]
   [18] invoke_in_world
      @ ./essentials.jl:923 [inlined]
   [19] require
      @ ./loading.jl:1746
   [20] eval
      @ ./boot.jl:385
   [21] #invokelatest#2
      @ ./essentials.jl:892
   [22] invokelatest
      @ ./essentials.jl:889
   [23] #114
      @ ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:303
   [24] run_work_thunk
      @ ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:70
   [25] run_work_thunk
      @ ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:79
   [26] #100
      @ ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:88
  during initialization of module Sleipnir

  ...and 2 more exceptions.

  Stacktrace:
    [1] sync_end(c::Channel{Any})
      @ Base ./task.jl:455
    [2] macro expansion
      @ ./task.jl:487 [inlined]
    [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
      @ Distributed ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/macros.jl:219
    [4] macro expansion
      @ ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/macros.jl:203 [inlined]
    [5] top-level scope
      @ ~/.julia/dev/Sleipnir/src/setup/config.jl:117
    [6] enable_multiprocessing(procs::Int64)
      @ Sleipnir ~/.julia/dev/Sleipnir/src/setup/config.jl:0
    [7] #Parameters#14
      @ ~/.julia/dev/Sleipnir/src/parameters/Parameters.jl:44 [inlined]
    [8] Parameters
      @ ~/.julia/dev/Sleipnir/src/parameters/Parameters.jl:33 [inlined]
    [9] params_constructor_default(; save_refs::Bool)
      @ Main ~/.julia/dev/Sleipnir/test/params_construction.jl:70
   [10] params_constructor_default()
      @ Main ~/.julia/dev/Sleipnir/test/params_construction.jl:62
   [11] macro expansion
      @ ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [12] top-level scope
      @ ~/.julia/dev/Sleipnir/test/runtests.jl:1573
   [13] include(fname::String)
      @ Base.MainInclude ./client.jl:494
   [14] top-level scope
      @ REPL[4]:1
   [15] eval
      @ ./boot.jl:385 [inlined]
   [16] eval_user_input(ast::Any, backend::REPL.REPLBackend, mod::Module)
      @ REPL ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:150
   [17] repl_backend_loop(backend::REPL.REPLBackend, get_module::Function)
      @ REPL ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:246
   [18] start_repl_backend(backend::REPL.REPLBackend, consumer::Any; get_module::Function)
      @ REPL ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:231
   [19] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool, backend::Any)
      @ REPL ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:389
   [20] run_repl(repl::REPL.AbstractREPL, consumer::Any)
      @ REPL ~/.julia/juliaup/julia-1.10.6+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:375
   [21] (::Base.var"#1013#1015"{Bool, Bool, Bool})(REPL::Module)
      @ Base ./client.jl:437
   [22] #invokelatest#2
      @ ./essentials.jl:892 [inlined]
   [23] invokelatest
      @ ./essentials.jl:889 [inlined]
   [24] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
      @ Base ./client.jl:421
   [25] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:338
   [26] _start()
      @ Base ./client.jl:557
Test Summary:                      | Error  Total   Time
Parameters constructors by default |     1      1  10.6s
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /Users/sapienza/.julia/dev/Sleipnir/test/runtests.jl:20

Maybe this error can be avoided in the same way that the error when doing the using Sleipnir was solved, but so far it seems that the error just propagated to a different part of the workflow.

[facusapienza21]

Is there a way to make a global config for this? It seems that @cjdoris you got into something with your first suggestion, I am wondering if there is a robust solution for this problem. Thanks!

[JordiBolibar]

I'm also experiencing the same problem.

The strangest thing is that we can make this work using Facu's instructions from above, but ONLY if the Julia session is exited in between the using OpenSSL_jll and using PythonCall; pyimport("ssl"). This makes it impossible to reproduce and fix this during the initialization of Sleipnir. If we don't exit the session in between it doesn't work anymore.

@cjdoris any idea on what could be the issue? How could we reproduce the behaviour we have working in the REPL in our Sleipnir.jl and init? Thanks in advance!