`OpenSSL_jll` changes behavior of python dependencies loaded via `PythonCall`

[lassepe]

Cross posting from: https://github.com/avik-pal/Wandb.jl/issues/39

TL;DR loading OpenSSL_jll before Wandb.jl causes Wandb.jl to error out with an SSL error. Loading them in the opposite order is fine.

Reproducer:

using OpenSSL_jll
using Wandb: Wandb

# trigger ssl error
function provoke_error(logger)
    logging_dict = Dict{String,Any}()
    logging_dict["samples"] = Wandb.Image(rand(2, 2))
    Wandb.log(logger, logging_dict; step=1)
end

logger = Wandb.WandbLogger(; project="test")
provoke_error(logger)

Error:

Click to expand error

```julia ERROR: LoadError: Python: CommError: Can't connect to HTTPS URL because the SSL module is not available. Python stacktrace: [1] _new_conn @ urllib3.connectionpool ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/urllib3/connectionpool.py:1059 [2] _get_conn @ urllib3.connectionpool ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/urllib3/connectionpool.py:294 [3] urlopen @ urllib3.connectionpool ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/urllib3/connectionpool.py:768 [4] send @ requests.adapters ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/requests/adapters.py:667 [5] send @ requests.sessions ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/requests/sessions.py:703 [6] request @ requests.sessions ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/requests/sessions.py:589 [7] post @ requests.sessions ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/requests/sessions.py:637 [8] execute @ wandb.sdk.lib.gql_request ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/lib/gql_request.py:58 [9] _get_result @ wandb_gql.client ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py:60 [10] execute @ wandb_gql.client ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py:52 [11] execute @ wandb.sdk.internal.internal_api ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/internal/internal_api.py:345 [12] __call__ @ wandb.sdk.lib.retry ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/lib/retry.py:131 [13] gql @ wandb.sdk.internal.internal_api ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/internal/internal_api.py:317 [14] server_info_introspection @ wandb.sdk.internal.internal_api ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/internal/internal_api.py:557 [15] wrapper @ wandb.apis.normalize ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/apis/normalize.py:41 [16] wrapper @ wandb.apis.normalize ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/apis/normalize.py:87 [17] max_cli_version @ wandb.sdk.internal.internal_api ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/internal/internal_api.py:870 [18] wrapper @ wandb.apis.normalize ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/apis/normalize.py:41 [19] wrapper @ wandb.apis.normalize ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/apis/normalize.py:73 [20] max_cli_version @ wandb.apis.internal ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/apis/internal.py:82 [21] _get_max_cli_version @ wandb.util ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/util.py:1710 [22] _server_accepts_artifact_path @ wandb.sdk.data_types.image ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/data_types/image.py:57 [23] bind_to_run @ wandb.sdk.data_types.image ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/data_types/image.py:402 [24] val_to_json @ wandb.sdk.data_types.utils ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/data_types/utils.py:164 [25] history_dict_to_json @ wandb.sdk.data_types.utils ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/data_types/utils.py:52 [26] publish_partial_history @ wandb.sdk.interface.interface ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/interface/interface.py:600 [27] _partial_history_callback @ wandb.sdk.wandb_run ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/wandb_run.py:1513 [28] _log @ wandb.sdk.wandb_run ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/wandb_run.py:1641 [29] log @ wandb.sdk.wandb_run ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/wandb_run.py:1877 [30] wrapper @ wandb.sdk.wandb_run ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/wandb_run.py:390 [31] wrapper_fn @ wandb.sdk.wandb_run ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/wandb_run.py:400 [32] wrapper @ wandb.sdk.wandb_run ~/worktree/BugReports/Wandb.jl-load-order/.CondaPkg/env/lib/python3.12/site-packages/wandb/sdk/wandb_run.py:449 Stacktrace: [1] pythrow() @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/err.jl:92 [2] errcheck @ ~/.julia/packages/PythonCall/S5MOg/src/Core/err.jl:10 [inlined] [3] pycallargs @ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:213 [inlined] [4] pycall(f::PythonCall.Core.Py, args::PythonCall.Core.Py; kwargs::@Kwargs{step::Int64}) @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:224 [5] pycall @ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:220 [inlined] [6] Py @ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:339 [inlined] [7] log(lg::Wandb.WandbLogger, logs::Dict{String, Any}; kwargs::@Kwargs{step::Int64}) @ Wandb ~/worktree/BugReports/Wandb.jl-load-order/dev/Wandb/src/main.jl:69 [8] log @ ~/worktree/BugReports/Wandb.jl-load-order/dev/Wandb/src/main.jl:69 [inlined] [9] provoke_error(logger::Wandb.WandbLogger) @ Main ~/worktree/BugReports/Wandb.jl-load-order/main.jl:8 [10] top-level scope @ ~/worktree/BugReports/Wandb.jl-load-order/main.jl:13 ```

[giordano]

When you mix libraries coming from different sources all odds are off.

[lassepe]

But is it expected that libs loaded via _jll are visible inside PythonCall dependencies? Is there no way to isolate those two environments?

[lassepe]

To be clear, I am not sure whether this "isolation" would be the _jll's responsibility or there needs to be a change on the side of PythonCall.jl --- just trying to build an understanding of where the issue needs to be filed.

[giordano]

But is it expected that libs loaded via _jll are visible inside PythonCall dependencies?

Yes, of course, it's all in the same process.

Is there no way to isolate those two environments?

There are no "two environments", there's a single process. That's how you move data seamlessly between "Julia code" and "python code".

[lassepe]

I see. I never realized that the python dependency would adopt the exact same process but it makes a lot of sense. I am still surprised that there's no way of name-spacing the libraries somehow, e.g. using some mechanism like dlmopen. If that is infeasible, please feel free to close this issue! Thank you for your swift response, @giordano!

[giordano]

e.g. using some mechanism like dlmopen

dlmopen was suggested at https://github.com/JuliaPackaging/Yggdrasil/pull/3801#issuecomment-974309260, but (a) it's glibc-specific so it doesn't solve anything on all other platforms (b) it has its own problems.