julia: enable all tests + stricter pinning

[ngam]

This is an issue to track progress and discussion regarding the two points below.

• We ought to conduct all tests available upstream (unless they run over 6 hours).

• Based on the issues we faced with libunwind and now openlibm, we should consider more exact pinning going forward.

---

• [x] I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Environment (conda list):

``` $ conda list ```

Details about conda and system ( conda info ):

``` $ conda info ```

[ngam]

@mkitti would this call suffice?

Base.runtests(tests=["all"]; ncores=ceil(Int, Sys.CPU_THREADS / 2),
              exit_on_error=false, revise=false, [seed])

Run the Julia unit tests listed in tests, which can be either a string or an array of strings, using ncores processors. If exit_on_error is false, when one test fails, all remaining tests in other files will still be run; they are otherwise discarded, when exit_on_error == true. If revise is true, the Revise package is used to load any modifications to Base or to the standard libraries before running the tests. If a seed is provided via the keyword argument, it is used to seed the global RNG in the context where the tests are run; otherwise the seed is chosen randomly.

https://docs.julialang.org/en/v1/stdlib/Test/#Testing-Base-Julia

[mkitti]

We also need to enable the doctests, the network tests, Pkg tests, and LibGit2/online

https://github.com/JuliaLang/julia/blob/98b485e0641a2e6017ae91acc9460c668d0e811a/test/choosetests.jl#L167

[mkitti]

We probably should talk to someone before turning all of this on. Also, it might be best to manually trigger these tests. The impression I was given earlier was that we should not do all of this all the time.

[ngam]

We probably should talk to someone before turning all of this on. Also, it might be best to manually trigger these tests. The impression I was given earlier was that we should not do all of this all the time.

It depends on how long it will take --- if it takes the build time from 1 hour to 2 hours, I wouldn't worry about talking to anyone else about it. But if we are going to push basically to 6 hours, then obviously, we will want to talk to people about it. Generally speaking, my personal impression is that the governance here is pretty loose and as long as you're within reasonable limits and not testing all the time with rebuilds (like we are doing now because of the activity), it should be okay.

When I enabled all the base tests, it didn't take significantly longer fwiw. I am not sure about the other tests (I don't even know how I would go about activating them)

Tagging @isuruf for a quick word, if any.

[mkitti]

The tests used to time out: https://github.com/conda-forge/julia-feedstock/pull/44#issuecomment-360880573 but maybe it had something to do CircleCI and the tests not being verbose.

[mkitti]

Here is the PR and comment when most of the tests were disabled, which is how I found it several months ago. https://github.com/conda-forge/julia-feedstock/pull/50#issuecomment-405004969

Anyways, let's just try it and see what happens.

[ngam]

See this: https://github.com/conda-forge/julia-feedstock/pull/157/commits/c2fd0f995717de10237654036bf9f36320d3f762

with results here (using half the ncores): https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=432645&view=results

can;t open, but this is a hint:

Azure Pipelines / julia-feedstock (osx osx64) cancelled 11 days ago in 1h 57m 15s

I cancelled the run basically after the scoresheet came out --- there were many failures.

Point is, only 2 hours.

[mkitti]

We should also add downstream testing.

@valeriupredoi, I see https://github.com/ESMValGroup/ESMValTool/blob/main/tests/integration/diagnostic.jl in the ESMValTool test suite. Are there any other Julia based test files?

[mkitti]

When we have to patch to disable tests, we should use @test_broken. This will tell us when the tests start working. https://github.com/conda-forge/julia-feedstock/blob/bd5133e708542ffdfb3ae5131636d3db3c58f4fa/recipe/patches/0002-disable-testing-Baz.baz-1.patch#L19

[valeriupredoi]

Hi @mkitti - that diagnostic.jl is just a script that we run part of the test suite to see if a Julia diagnostic can be run by ESMValTool - don't think of it as a test diagnostic, we use the term "diagnostic" to refer to our climate data diagnostics, and not as a system tool. This gets run in a subprocess with julia diagnostic.jl so just like any other Julia script in an environment provided with a julia executable :+1: And no, I believe this is the only functional test we run with Julia :beer:

[ngam]

osx-64/julia-1.7.1-h132cb31_2.tar.bz2

arpack >=3.7.0,<3.7.1.0a0, 
curl >=7.80.0,<8.0a0, 
git, 
gmp >=6.2.1,<7.0a0, 
libcxx >=11.1.0, 
libgfortran 5.*, 
libgfortran5 >=9.3.0, 
libnghttp2 >=1.43.0,<2.0a0, 
libosxunwind, 
libssh2 >=1.10.0,<2.0a0, 
libutf8proc >=2.7.0,<3.0a0, 
libzlib >=1.2.11,<1.3.0a0, 
mpfr >=4.1.0,<5.0a0, 
openblas-ilp64, 
openlibm <0.8.0, 
p7zip, pcre2 >=10.37,<10.38.0a0, 
suitesparse >=5.10.1,<6.0a0, 
zlib >=1.2.11,<1.3.0a0

linux-64/julia-1.7.1-h989b2f6_2.tar.bz2

arpack >=3.7.0,<3.7.1.0a0, 
curl >=7.80.0,<8.0a0, 
git, 
gmp >=6.2.1,<7.0a0, 
libgcc-ng >=9.4.0, 
libgfortran-ng, 
libgfortran5 >=9.4.0, 
libgit2 >=1.3.0,<1.4.0a0, 
libnghttp2 >=1.43.0,<2.0a0, 
libssh2 >=1.10.0,<2.0a0, 
libstdcxx-ng >=9.4.0, 
libunwind >=1.5.0,<1.6.0a0, 
libutf8proc >=2.7.0,<3.0a0, 
libzlib >=1.2.11,<1.3.0a0, 
mbedtls, 
mpfr >=4.1.0,<5.0a0, 
openblas-ilp64, 
openlibm <0.8.0, 
p7zip, 
pcre2 >=10.37,<10.38.0a0, 
suitesparse >=5.10.1,<6.0a0, 
zlib >=1.2.11,<1.3.0a0

[ngam]

mbedtls constraints: https://github.com/JuliaLang/julia/issues/43624#issuecomment-1004037620

[mkitti]

I'm not sure how well you can derive constraints from that issue. Julia 1.7.1 is hard pinned to 2.24. There is significant ABI breakage from between Mbed TLS 2.24 and 2.28.

https://github.com/JuliaPackaging/Yggdrasil/pull/4180

[ngam]

I'm not sure how well you can derive constraints from that issue

Wasn't thinking of that issue as the source. I just wanted to note that if we are to constrain mbedtls, we will need to go through all these (and your CVE-linked issue as well). Just bookmarking.

[ngam]

FYI:

the global pins usually appear in here: https://github.com/conda-forge/julia-feedstock/blob/master/.ci_support/linux_64_.yaml

You can edit them in here: https://github.com/conda-forge/julia-feedstock/blob/master/recipe/conda_build_config.yaml

For example, editing conda_build_config.yaml by adding:

curl: 
  - 7
...
pin_run_as_build:
...
    curl:
      max_pin: 7.80

will result in linux_64_.yaml and osx_64_.yaml:

curl:
- '7'
...
pin_run_as_build:
...
  curl:
    max_pin: '7.80'

and so on.

We should this for global pins, basically just pinning to the latest release (so that it is tested by us) and we can just add max pins in meta for the rest. The full list is in a previous comment.

[ngam]

@mkitti, should we pursue this max pinning sooner rather than later? We simply cannot afford to wait for the global pinning efforts and we will likely have other problems pretty soon.

[ngam]

https://github.com/conda-forge/curl-feedstock/pull/101#issuecomment-1007639506

[mkitti]

I'm tempted to pin the configurations that we actually test. We would then periodically, weekly perhaps, relax the pins, retest, and then repin before merging again.

For official builds, perhaps someone should package juliaup: https://github.com/JuliaLang/juliaup We could then provide the activate and deactivate scripts with that as well.

[ngam]

I'm tempted to pin the configurations that we actually test. We would then periodically, weekly perhaps, relax the pins, retest, and then repin before merging again.

I understand... but if you look above, you will see some pretty decent global pinning is going on. Generally, conda/mamba will resolve to the tested version or very near it. The only problem we faced thus far is when packages upgrade, and hence we may get all we need with a simple max pinning.

I still stand by my idea of rebuilding the exact same julia (or repackaging) so that's definitely something I am interested in pursuing (see #175)

[mkitti]

I think juliaup offers a mechanism for systematic repackaging.

[ngam]

It doesn't look like we are actually using the conda-forge suitesparse (among many other deps) in the build despite the explicit instruction... a small segment of the build:

make[5]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/CHOLMOD/Lib'
make[4]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/CHOLMOD/Lib'
make[3]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/CHOLMOD'
make[3]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL'
make[4]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
make[5]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
ar: creating libldl.a
a - ldl.o
a - ldll.o
make[5]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
make[4]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
make[3]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL'
make[3]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/KLU'
make[4]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/KLU/Lib'
make[5]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/KLU/Lib'
ar: creating libklu.a
a - klu_free_symbolic.o
a - klu_defaults.o
a - klu_analyze_given.o
a - klu_analyze.o
a - klu_memory.o
a - klu_l_free_symbolic.o
a - klu_l_defaults.o
a - klu_l_analyze_given.o
a - klu_l_analyze.o
a - klu_l_memory.o
a - klu_d.o
a - klu_d_kernel.o
a - klu_d_dump.o
a - klu_d_factor.o
a - klu_d_free_numeric.o
a - klu_d_solve.o
a - klu_d_scale.o
a - klu_d_refactor.o
a - klu_d_tsolve.o
a - klu_d_diagnostics.o
a - klu_d_sort.o
a - klu_d_extract.o
a - klu_z.o
a - klu_z_kernel.o
a - klu_z_dump.o
a - klu_z_factor.o

and it keeps going in and out of suitesparse and other deps --- not sure if it is rebuilding stuff or doing something different.

We need to investgiate this further. Below are some of the warnings at the end:

WARNING (julia,lib/julia/libllvmcalltest.so): $RPATH/libLLVM-13jl.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
...
WARNING (julia,lib/julia/sys.so): $RPATH/libjulia-internal.so.1 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
...
WARNING (julia): run-exports library package conda-forge::pcre2-10.37-h032f7d1_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::libgit2-1.3.0-haabb1ae_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::libgfortran5-11.2.0-h5c6108e_11 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::arpack-3.7.0-hdefa2d7_2 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::mpfr-4.1.0-h9202a9a_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::p7zip-16.02-he1b5a44_1000 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::zlib-1.2.11-h36c2ea0_1013 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::gmp-6.2.1-h58526e2_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::mbedtls-3.1.0-h9c3ff4c_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::libnghttp2-1.43.0-ha19adfc_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::libssh2-1.10.0-ha35d2d1_2 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::openlibm-0.7.5-h7f98852_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

@isuruf + @beckermr could you briefly check this out and let us know if you think these warnings are worth following up?

[ngam]

Note for this one, WARNING (julia,lib/julia/libllvmcalltest.so): $RPATH/libLLVM-13jl.so not found in packages, we actually intentionally patch libLLVM-13jl.so to libLLVM.so. So it could be simply we need to patch it somewhere else too to ensure it is actually all good.

[mkitti]

The option should be USE_SYSTEM_LIBSUITESPARSE

[ngam]

Are these the legit options or are there more? https://github.com/JuliaLang/julia/blob/cd81054bae301ccf3a0fe7e0d8f60f167f1203a0/Makefile#L161-L222

# public libraries, that are installed in $(prefix)/lib
JL_TARGETS := julia
ifeq ($(BUNDLE_DEBUG_LIBS),1)
JL_TARGETS += julia-debug
endif

# private libraries, that are installed in $(prefix)/lib/julia
JL_PRIVATE_LIBS-0 := libccalltest libllvmcalltest libjulia-internal libjulia-codegen
ifeq ($(BUNDLE_DEBUG_LIBS),1)
JL_PRIVATE_LIBS-0 += libjulia-internal-debug libjulia-codegen-debug
endif
ifeq ($(USE_GPL_LIBS), 1)
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBSUITESPARSE) += libamd libbtf libcamd libccolamd libcholmod libcolamd libklu libldl librbio libspqr libsuitesparseconfig libumfpack
endif
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBBLASTRAMPOLINE) += libblastrampoline
JL_PRIVATE_LIBS-$(USE_SYSTEM_PCRE) += libpcre2-8
JL_PRIVATE_LIBS-$(USE_SYSTEM_DSFMT) += libdSFMT
JL_PRIVATE_LIBS-$(USE_SYSTEM_GMP) += libgmp libgmpxx
JL_PRIVATE_LIBS-$(USE_SYSTEM_MPFR) += libmpfr
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBSSH2) += libssh2
JL_PRIVATE_LIBS-$(USE_SYSTEM_NGHTTP2) += libnghttp2
JL_PRIVATE_LIBS-$(USE_SYSTEM_MBEDTLS) += libmbedtls libmbedcrypto libmbedx509
JL_PRIVATE_LIBS-$(USE_SYSTEM_CURL) += libcurl
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBGIT2) += libgit2
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBUV) += libuv
ifeq ($(OS),WINNT)
JL_PRIVATE_LIBS-$(USE_SYSTEM_ZLIB) += zlib
else
JL_PRIVATE_LIBS-$(USE_SYSTEM_ZLIB) += libz
endif
ifeq ($(USE_LLVM_SHLIB),1)
JL_PRIVATE_LIBS-$(USE_SYSTEM_LLVM) += libLLVM libLLVM-13jl
endif
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBUNWIND) += libunwind

ifeq ($(USE_SYSTEM_LIBM),0)
JL_PRIVATE_LIBS-$(USE_SYSTEM_OPENLIBM) += libopenlibm
endif

JL_PRIVATE_LIBS-$(USE_SYSTEM_BLAS) += $(LIBBLASNAME)
ifneq ($(LIBLAPACKNAME),$(LIBBLASNAME))
JL_PRIVATE_LIBS-$(USE_SYSTEM_LAPACK) += $(LIBLAPACKNAME)
endif

JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libgfortran libquadmath libstdc++ libgcc_s libgomp libssp libatomic
ifeq ($(OS),Darwin)
JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libc++
endif
ifeq ($(OS),WINNT)
JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libwinpthread
else
JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libpthread
endif

ifeq ($(OS),Darwin)
ifeq ($(USE_SYSTEM_BLAS),1)
ifeq ($(USE_SYSTEM_LAPACK),0)
JL_PRIVATE_LIBS-0 += libgfortblas
endif
endif
endif

[ngam]

Any idea why we aren't using our own libuv? I understand we have longstanding issues iwth libgit2 and mbedtls, and llvm is a hot mess. But libuv --- what's going on there? Also USE_SYSTEM_CSL: https://github.com/JuliaLang/julia/issues?q=USE_SYSTEM_CSL%3D1

export EXTRA_MAKEFLAGS="" 
if [[ "${target_platform}" == osx-* ]]; then
    export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=0 USE_SYSTEM_MBEDTLS=0"
elif [[ "${target_platform}" == linux-* ]]; then
    export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_MBEDTLS=1"
fi

...

make -j${CPU_COUNT} prefix=${PREFIX} sysconfigdir=${PREFIX}/etc \
 LIBBLAS=-lopenblas64_ LIBBLASNAME=libopenblas64_ LIBLAPACK=-lopenblas64_ LIBLAPACKNAME=libopenblas64_ \
 USE_SYSTEM_ARPACK=1 \
 USE_SYSTEM_BLAS=1 \
 USE_BLAS64=1 \
 USE_SYSTEM_CURL=1 \
 USE_SYSTEM_GMP=1 \
 USE_SYSTEM_LAPACK=1 \
 USE_SYSTEM_LIBSSH2=1 \
 USE_SYSTEM_LLVM=0 \
 USE_SYSTEM_MPFR=1 \
 USE_SYSTEM_OPENLIBM=1 \
 USE_SYSTEM_PATCHELF=1 \
 USE_SYSTEM_PCRE=1 \
 USE_SYSTEM_SUITESPARSE=1 \
 USE_SYSTEM_CSL=0 \
 USE_SYSTEM_LIBUNWIND=1 \
 USE_SYSTEM_LIBUV=0 \
 USE_SYSTEM_UTF8PROC=1 \
 USE_SYSTEM_NGHTTP2=1 \
 USE_SYSTEM_ZLIB=1 \
 USE_SYSTEM_P7ZIP=1 \
 ${EXTRA_MAKEFLAGS} \
 TAGGED_RELEASE_BANNER="https://github.com/conda-forge/julia-feedstock" \
 CC=$CC CXX=$CXX FC=$FC \
 install

[ngam]

Note for this one, WARNING (julia,lib/julia/libllvmcalltest.so): $RPATH/libLLVM-13jl.so not found in packages, we actually intentionally patch libLLVM-13jl.so to libLLVM.so. So it could be simply we need to patch it somewhere else too to ensure it is actually all good.

hmmmmm: JL_PRIVATE_LIBS-$(USE_SYSTEM_LLVM) += libLLVM libLLVM-13jl

[ngam]

@mkitti, let's review the archlinux patch more closely: https://github.com/conda-forge/julia-feedstock/blob/master/recipe/patches/julia-hardcoded-libs.patch, I think it may be doing some harm (e.g. we shouldn't be patching the LLVM deps, since we are not using the system one)

[ngam]

@mkitti --- help me understand this. So we are building against openblas, openlibm, suitesparse, and zlib (to name a few) and yet if you end up adding a package inside julia, you will essentially be re-providing these from julia (see below):

In julia's Pkg:

  [4536629a] + OpenBLAS_jll v0.3.17+2
  [05823500] + OpenLibm_jll v0.7.5+0
  [bea87d4a] + SuiteSparse_jll v5.10.1+0
  [83775a58] + Zlib_jll v1.2.12+1

In conda's list:

openblas-ilp64            0.3.18          openmp_ha601604_0    conda-forge
openlibm                  0.7.5                h0d85af4_0    conda-forge
suitesparse               5.10.1               h7aff33d_1    conda-forge
zlib                      1.2.11            h9173be1_1013    conda-forge

So... what's the point? We are not really linking properly against system utilities at the end of the day, or are we?

(jdev) ngam@mp15:~$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-DEV.1262 (2021-09-20)
 _/ |\__'_|_|_|\__'_|  |  HEAD/9ea2cc48b* (fork: 1 commits, 1 day)
|__/                   |

julia> versioninfo()
Julia Version 1.8.0-DEV.1262
Commit 9ea2cc48b* (2021-09-20 14:56 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.0 (ORCJIT, ivybridge)
Environment:
  JULIA_DEPOT_PATH_BACKUP = 
  JULIA_PROJECT_BACKUP = 
  JULIA_LOAD_PATH_BACKUP = 
  JULIA_DEPOT_PATH = /Users/ngam/.Mambaforge-MacOSX-x86_64/envs/jdev/share/julia:
  JULIA_PROJECT = @jdev
  JULIA_LOAD_PATH = @:@jdev:@stdlib
  JULIA_SSL_CA_ROOTS_PATH_BACKUP = 
  JULIA_SSL_CA_ROOTS_PATH = /Users/ngam/.Mambaforge-MacOSX-x86_64/envs/jdev/ssl/cacert.pem

julia> 

julia> 

(@jdev) pkg> add sundials
    Updating registry at `~/.Mambaforge-MacOSX-x86_64/envs/jdev/share/julia/registries/General.toml`
ERROR: The following package names could not be resolved:
 * sundials (not found in project, manifest or registry)

(@jdev) pkg> add Sundials
   Resolving package versions...
   Installed MuladdMacro ────────────────────── v0.2.2
   Installed SIMDDualNumbers ────────────────── v0.1.0
   Installed IrrationalConstants ────────────── v0.1.1
   Installed BitTwiddlingConvenienceFunctions ─ v0.1.1
   Installed Adapt ──────────────────────────── v3.3.3
   Installed SIMDTypes ──────────────────────── v0.1.0
   Installed StrideArraysCore ───────────────── v0.2.9
   Installed Rmath ──────────────────────────── v0.7.0
   Installed Sundials ───────────────────────── v4.9.1
   Installed LayoutPointers ─────────────────── v0.1.4
   Installed TreeViews ──────────────────────── v0.3.0
   Installed DiffRules ──────────────────────── v1.9.0
   Installed DiffEqBase ─────────────────────── v6.80.0
   Installed SciMLBase ──────────────────────── v1.25.0
   Installed OffsetArrays ───────────────────── v1.10.8
   Installed VectorizationBase ──────────────── v0.21.23
   Installed Preferences ────────────────────── v1.2.3
   Installed StatsFuns ──────────────────────── v0.9.14
   Installed TableTraits ────────────────────── v1.0.1
   Installed DiffResults ────────────────────── v1.0.3
   Installed Hwloc_jll ──────────────────────── v2.7.0+0
   Installed NonlinearSolve ─────────────────── v0.3.13
   Installed CPUSummary ─────────────────────── v0.1.6
   Installed IfElse ─────────────────────────── v0.1.1
   Installed Tables ─────────────────────────── v1.6.1
   Installed PDMats ─────────────────────────── v0.11.5
   Installed SpecialFunctions ───────────────── v2.0.0
   Installed Setfield ───────────────────────── v0.8.1
   Installed DataAPI ────────────────────────── v1.9.0
   Installed LabelledArrays ─────────────────── v1.7.0
   Installed RecipesBase ────────────────────── v1.2.1
   Installed JLLWrappers ────────────────────── v1.3.0
   Installed DEDataArrays ───────────────────── v0.2.0
   Installed IterativeSolvers ───────────────── v0.9.2
   Installed Polyester ──────────────────────── v0.6.2
   Installed NaNMath ────────────────────────── v0.3.6
   Installed IteratorInterfaceExtensions ────── v1.0.0
   Installed StaticArrays ───────────────────── v1.3.1
   Installed ConstructionBase ───────────────── v1.3.0
   Installed HostCPUFeatures ────────────────── v0.1.5
   Installed DataValueInterfaces ────────────── v1.0.0
   Installed Hwloc ──────────────────────────── v2.0.0
   Installed LoopVectorization ──────────────── v0.12.101
   Installed Sundials_jll ───────────────────── v5.2.1+0
   Installed OrderedCollections ─────────────── v1.4.1
   Installed CEnum ──────────────────────────── v0.4.1
   Installed ManualMemory ───────────────────── v0.1.6
   Installed ThreadingUtilities ─────────────── v0.4.7
   Installed RecursiveArrayTools ────────────── v2.24.0
   Installed FunctionWrappers ───────────────── v1.1.2
   Installed ChainRulesCore ─────────────────── v1.11.4
   Installed PolyesterWeave ─────────────────── v0.1.2
   Installed ArrayInterface ─────────────────── v3.2.2
   Installed QuadGK ─────────────────────────── v2.4.2
   Installed ForwardDiff ────────────────────── v0.10.24
   Installed FillArrays ─────────────────────── v0.12.7
   Installed Rmath_jll ──────────────────────── v0.3.0+0
   Installed Reexport ───────────────────────── v1.2.2
   Installed TriangularSolve ────────────────── v0.1.9
   Installed PreallocationTools ─────────────── v0.2.1
   Installed DataStructures ─────────────────── v0.18.11
   Installed CommonSubexpressions ───────────── v0.3.0
   Installed LogExpFunctions ────────────────── v0.3.6
   Installed FastBroadcast ──────────────────── v0.1.12
   Installed Requires ───────────────────────── v1.2.0
   Installed ZygoteRules ────────────────────── v0.2.2
   Installed CloseOpenIntervals ─────────────── v0.1.4
   Installed MacroTools ─────────────────────── v0.5.9
   Installed Static ─────────────────────────── v0.4.1
   Installed SLEEFPirates ───────────────────── v0.6.28
   Installed StatsAPI ───────────────────────── v1.2.0
   Installed Compat ─────────────────────────── v3.41.0
   Installed UnPack ─────────────────────────── v1.0.2
   Installed Parameters ─────────────────────── v0.12.3
   Installed OpenSpecFun_jll ────────────────── v0.5.5+0
   Installed InverseFunctions ───────────────── v0.1.2
   Installed CommonSolve ────────────────────── v0.2.0
   Installed SortingAlgorithms ──────────────── v1.0.1
   Installed RecursiveFactorization ─────────── v0.2.7
   Installed Distributions ──────────────────── v0.25.37
   Installed Missings ───────────────────────── v1.0.2
   Installed DensityInterface ───────────────── v0.4.0
   Installed ChangesOfVariables ─────────────── v0.1.2
   Installed StatsBase ──────────────────────── v0.33.14
   Installed FiniteDiff ─────────────────────── v2.9.0
   Installed DocStringExtensions ────────────── v0.8.6
  Downloaded artifact: Hwloc
  Downloaded artifact: Sundials
  Downloaded artifact: Rmath
  Downloaded artifact: OpenSpecFun
    Updating `~/.Mambaforge-MacOSX-x86_64/envs/jdev/share/julia/environments/jdev/Project.toml`
  [c3572dad] + Sundials v4.9.1
    Updating `~/.Mambaforge-MacOSX-x86_64/envs/jdev/share/julia/environments/jdev/Manifest.toml`
  [79e6a3ab] + Adapt v3.3.3
⌅ [4fba245c] + ArrayInterface v3.2.2
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.1
  [fa961155] + CEnum v0.4.1
  [2a0fbf3d] + CPUSummary v0.1.6
  [d360d2e6] + ChainRulesCore v1.11.4
  [9e997f8a] + ChangesOfVariables v0.1.2
  [fb6a15b2] + CloseOpenIntervals v0.1.4
  [38540f10] + CommonSolve v0.2.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.41.0
  [187b0558] + ConstructionBase v1.3.0
  [754358af] + DEDataArrays v0.2.0
  [9a962f9c] + DataAPI v1.9.0
  [864edb3b] + DataStructures v0.18.11
  [e2d170a0] + DataValueInterfaces v1.0.0
  [b429d917] + DensityInterface v0.4.0
  [2b5f629d] + DiffEqBase v6.80.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.9.0
  [31c24e10] + Distributions v0.25.37
  [ffbed154] + DocStringExtensions v0.8.6
  [7034ab61] + FastBroadcast v0.1.12
  [1a297f60] + FillArrays v0.12.7
  [6a86dc24] + FiniteDiff v2.9.0
  [f6369f11] + ForwardDiff v0.10.24
  [069b7b12] + FunctionWrappers v1.1.2
  [3e5b6fbb] + HostCPUFeatures v0.1.5
  [0e44f5e4] + Hwloc v2.0.0
  [615f187c] + IfElse v0.1.1
  [3587e190] + InverseFunctions v0.1.2
  [92d709cd] + IrrationalConstants v0.1.1
  [42fd0dbc] + IterativeSolvers v0.9.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [2ee39098] + LabelledArrays v1.7.0
  [10f19ff3] + LayoutPointers v0.1.4
  [2ab3a3ac] + LogExpFunctions v0.3.6
  [bdcacae8] + LoopVectorization v0.12.101
  [1914dd2f] + MacroTools v0.5.9
  [d125e4d3] + ManualMemory v0.1.6
  [e1d29d7a] + Missings v1.0.2
  [46d2c3a1] + MuladdMacro v0.2.2
  [77ba4419] + NaNMath v0.3.6
  [8913a72c] + NonlinearSolve v0.3.13
  [6fe1bfb0] + OffsetArrays v1.10.8
  [bac558e1] + OrderedCollections v1.4.1
  [90014a1f] + PDMats v0.11.5
  [d96e819e] + Parameters v0.12.3
  [f517fe37] + Polyester v0.6.2
  [1d0040c9] + PolyesterWeave v0.1.2
  [d236fae5] + PreallocationTools v0.2.1
  [21216c6a] + Preferences v1.2.3
  [1fd47b50] + QuadGK v2.4.2
  [3cdcf5f2] + RecipesBase v1.2.1
  [731186ca] + RecursiveArrayTools v2.24.0
  [f2c3362d] + RecursiveFactorization v0.2.7
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.2.0
  [79098fc4] + Rmath v0.7.0
  [3cdde19b] + SIMDDualNumbers v0.1.0
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.28
  [0bca4576] + SciMLBase v1.25.0
  [efcf1570] + Setfield v0.8.1
  [a2af1166] + SortingAlgorithms v1.0.1
  [276daf66] + SpecialFunctions v2.0.0
⌅ [aedffcd0] + Static v0.4.1
  [90137ffa] + StaticArrays v1.3.1
  [82ae8749] + StatsAPI v1.2.0
  [2913bbd2] + StatsBase v0.33.14
  [4c63d2b9] + StatsFuns v0.9.14
  [7792a7ef] + StrideArraysCore v0.2.9
  [c3572dad] + Sundials v4.9.1
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.6.1
  [8290d209] + ThreadingUtilities v0.4.7
  [a2a6695c] + TreeViews v0.3.0
  [d5829a12] + TriangularSolve v0.1.9
  [3a884ed6] + UnPack v1.0.2
  [3d5dd08c] + VectorizationBase v0.21.23
  [700de1a5] + ZygoteRules v0.2.2
  [e33a78d0] + Hwloc_jll v2.7.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [f50d1b31] + Rmath_jll v0.3.0+0
  [fb77eaff] + Sundials_jll v5.2.1+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v0.5.0+0
  [deac9b47] + LibCURL_jll v7.73.0+4
  [29816b5a] + LibSSH2_jll v1.9.1+2
  [c8ffd9c3] + MbedTLS_jll v2.24.0+2
  [14a3606d] + MozillaCACerts_jll v2020.7.22
  [4536629a] + OpenBLAS_jll v0.3.17+2
  [05823500] + OpenLibm_jll v0.7.5+0
  [bea87d4a] + SuiteSparse_jll v5.10.1+0
  [83775a58] + Zlib_jll v1.2.12+1
  [8e850b90] + libblastrampoline_jll v3.1.0+0
  [8e850ede] + nghttp2_jll v1.41.0+1
  [3f19e933] + p7zip_jll v16.2.1+1
        Info Packages marked with ⌅ have new versions available but cannot be upgraded. To see why use `status --outdated`
Precompiling project...
  94 dependencies successfully precompiled in 120 seconds

(@jdev) pkg> 

[mkitti]

You installed Sundials which depends on Sundials_jll, which is a binary package that has binary dependencies.

If you really want a conda-forge version of this, you need to produce a version of Sundials_jll that uses conda-forge binaries and provides it's dependencies.

[mkitti]

The structure of the JLLs is fairly simple.

I made a sample of how you can make a mock one that would point to conda-forge binaries for the libcxxwrap_julia_jll https://github.com/mkitti/libcxxwrap_julia_jll_mock/blob/cf/src/libcxxwrap_julia_jll.jl

[mkitti]

The other mechanism is to use the existing Sundials_jll, but use overrides: https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artifact-locations

[ngam]

Okay, but essentially by default we are not seeing the benefit of our linking julia to system utilities? I understand your points about further customization, but we simply just cannot reproduce the whole julia ecosystem in conda-forge. We need to have all these communicate effectively, which seems like a losing battle. So two questions:

1. When is linking to system utilities (like we do) likely useful?
2. Is this the default for most packages like Sundials --- e.g. what if I install Cuda or any other random package, would I then get its associated binaries by default as well?

More interestingly, this doesn't seem to happen when you get julia from their binaries:

ngam@mp15:~$ /usr/local/bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.1 (2021-12-22)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, ivybridge)

(@v1.7) pkg> add Sundials
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed MuladdMacro ────────────────────── v0.2.2
   Installed BitTwiddlingConvenienceFunctions ─ v0.1.1
   Installed IrrationalConstants ────────────── v0.1.1
   Installed TreeViews ──────────────────────── v0.3.0
   Installed DiffRules ──────────────────────── v1.9.0
   Installed SIMDTypes ──────────────────────── v0.1.0
   Installed OffsetArrays ───────────────────── v1.10.8
   Installed Adapt ──────────────────────────── v3.3.3
   Installed SIMDDualNumbers ────────────────── v0.1.0
   Installed StrideArraysCore ───────────────── v0.2.9
   Installed Rmath ──────────────────────────── v0.7.0
   Installed Preferences ────────────────────── v1.2.3
   Installed StatsFuns ──────────────────────── v0.9.14
   Installed LayoutPointers ─────────────────── v0.1.4
   Installed DiffEqBase ─────────────────────── v6.80.0
   Installed VectorizationBase ──────────────── v0.21.23
   Installed Sundials ───────────────────────── v4.9.1
   Installed SciMLBase ──────────────────────── v1.25.0
   Installed NonlinearSolve ─────────────────── v0.3.13
   Installed Hwloc_jll ──────────────────────── v2.7.0+0
   Installed DiffResults ────────────────────── v1.0.3
   Installed IfElse ─────────────────────────── v0.1.1
   Installed CPUSummary ─────────────────────── v0.1.6
   Installed PDMats ─────────────────────────── v0.11.5
   Installed DataAPI ────────────────────────── v1.9.0
   Installed Tables ─────────────────────────── v1.6.1
   Installed Setfield ───────────────────────── v0.8.1
   Installed SpecialFunctions ───────────────── v2.0.0
   Installed TableTraits ────────────────────── v1.0.1
   Installed StaticArrays ───────────────────── v1.3.1
   Installed LabelledArrays ─────────────────── v1.7.0
   Installed RecipesBase ────────────────────── v1.2.1
   Installed LoopVectorization ──────────────── v0.12.101
   Installed JLLWrappers ────────────────────── v1.3.0
   Installed DEDataArrays ───────────────────── v0.2.0
   Installed NaNMath ────────────────────────── v0.3.6
   Installed Polyester ──────────────────────── v0.6.2
   Installed ConstructionBase ───────────────── v1.3.0
   Installed IteratorInterfaceExtensions ────── v1.0.0
   Installed IterativeSolvers ───────────────── v0.9.2
   Installed Hwloc ──────────────────────────── v2.0.0
   Installed HostCPUFeatures ────────────────── v0.1.5
   Installed DataValueInterfaces ────────────── v1.0.0
   Installed Sundials_jll ───────────────────── v5.2.1+0
   Installed ThreadingUtilities ─────────────── v0.4.7
   Installed FunctionWrappers ───────────────── v1.1.2
   Installed OrderedCollections ─────────────── v1.4.1
   Installed ManualMemory ───────────────────── v0.1.6
   Installed RecursiveArrayTools ────────────── v2.24.0
   Installed CEnum ──────────────────────────── v0.4.1
   Installed ChainRulesCore ─────────────────── v1.11.4
   Installed PolyesterWeave ─────────────────── v0.1.2
   Installed QuadGK ─────────────────────────── v2.4.2
   Installed Rmath_jll ──────────────────────── v0.3.0+0
   Installed TriangularSolve ────────────────── v0.1.9
   Installed FillArrays ─────────────────────── v0.12.7
   Installed ArrayInterface ─────────────────── v3.2.2
   Installed ForwardDiff ────────────────────── v0.10.24
   Installed Reexport ───────────────────────── v1.2.2
   Installed PreallocationTools ─────────────── v0.2.1
   Installed CommonSubexpressions ───────────── v0.3.0
   Installed Requires ───────────────────────── v1.2.0
   Installed ZygoteRules ────────────────────── v0.2.2
   Installed FastBroadcast ──────────────────── v0.1.12
   Installed LogExpFunctions ────────────────── v0.3.6
   Installed CloseOpenIntervals ─────────────── v0.1.4
   Installed DataStructures ─────────────────── v0.18.11
   Installed MacroTools ─────────────────────── v0.5.9
   Installed Distributions ──────────────────── v0.25.37
   Installed Static ─────────────────────────── v0.4.1
   Installed OpenSpecFun_jll ────────────────── v0.5.5+0
   Installed StatsAPI ───────────────────────── v1.2.0
   Installed Compat ─────────────────────────── v3.41.0
   Installed UnPack ─────────────────────────── v1.0.2
   Installed CommonSolve ────────────────────── v0.2.0
   Installed InverseFunctions ───────────────── v0.1.2
   Installed SLEEFPirates ───────────────────── v0.6.28
   Installed Parameters ─────────────────────── v0.12.3
   Installed RecursiveFactorization ─────────── v0.2.7
   Installed SortingAlgorithms ──────────────── v1.0.1
   Installed Missings ───────────────────────── v1.0.2
   Installed FiniteDiff ─────────────────────── v2.9.0
   Installed DensityInterface ───────────────── v0.4.0
   Installed ChangesOfVariables ─────────────── v0.1.2
   Installed StatsBase ──────────────────────── v0.33.14
   Installed DocStringExtensions ────────────── v0.8.6
  Downloaded artifact: Hwloc
  Downloaded artifact: Sundials
  Downloaded artifact: Rmath
  Downloaded artifact: OpenSpecFun
    Updating `~/.julia/environments/v1.7/Project.toml`
  [c3572dad] + Sundials v4.9.1
    Updating `~/.julia/environments/v1.7/Manifest.toml`
  [79e6a3ab] + Adapt v3.3.3
  [4fba245c] + ArrayInterface v3.2.2
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.1
  [fa961155] + CEnum v0.4.1
  [2a0fbf3d] + CPUSummary v0.1.6
  [d360d2e6] + ChainRulesCore v1.11.4
  [9e997f8a] + ChangesOfVariables v0.1.2
  [fb6a15b2] + CloseOpenIntervals v0.1.4
  [38540f10] + CommonSolve v0.2.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.41.0
  [187b0558] + ConstructionBase v1.3.0
  [754358af] + DEDataArrays v0.2.0
  [9a962f9c] + DataAPI v1.9.0
  [864edb3b] + DataStructures v0.18.11
  [e2d170a0] + DataValueInterfaces v1.0.0
  [b429d917] + DensityInterface v0.4.0
  [2b5f629d] + DiffEqBase v6.80.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.9.0
  [31c24e10] + Distributions v0.25.37
  [ffbed154] + DocStringExtensions v0.8.6
  [7034ab61] + FastBroadcast v0.1.12
  [1a297f60] + FillArrays v0.12.7
  [6a86dc24] + FiniteDiff v2.9.0
  [f6369f11] + ForwardDiff v0.10.24
  [069b7b12] + FunctionWrappers v1.1.2
  [3e5b6fbb] + HostCPUFeatures v0.1.5
  [0e44f5e4] + Hwloc v2.0.0
  [615f187c] + IfElse v0.1.1
  [3587e190] + InverseFunctions v0.1.2
  [92d709cd] + IrrationalConstants v0.1.1
  [42fd0dbc] + IterativeSolvers v0.9.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [2ee39098] + LabelledArrays v1.7.0
  [10f19ff3] + LayoutPointers v0.1.4
  [2ab3a3ac] + LogExpFunctions v0.3.6
  [bdcacae8] + LoopVectorization v0.12.101
  [1914dd2f] + MacroTools v0.5.9
  [d125e4d3] + ManualMemory v0.1.6
  [e1d29d7a] + Missings v1.0.2
  [46d2c3a1] + MuladdMacro v0.2.2
  [77ba4419] + NaNMath v0.3.6
  [8913a72c] + NonlinearSolve v0.3.13
  [6fe1bfb0] + OffsetArrays v1.10.8
  [bac558e1] + OrderedCollections v1.4.1
  [90014a1f] + PDMats v0.11.5
  [d96e819e] + Parameters v0.12.3
  [f517fe37] + Polyester v0.6.2
  [1d0040c9] + PolyesterWeave v0.1.2
  [d236fae5] + PreallocationTools v0.2.1
  [21216c6a] + Preferences v1.2.3
  [1fd47b50] + QuadGK v2.4.2
  [3cdcf5f2] + RecipesBase v1.2.1
  [731186ca] + RecursiveArrayTools v2.24.0
  [f2c3362d] + RecursiveFactorization v0.2.7
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.2.0
  [79098fc4] + Rmath v0.7.0
  [3cdde19b] + SIMDDualNumbers v0.1.0
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.28
  [0bca4576] + SciMLBase v1.25.0
  [efcf1570] + Setfield v0.8.1
  [a2af1166] + SortingAlgorithms v1.0.1
  [276daf66] + SpecialFunctions v2.0.0
  [aedffcd0] + Static v0.4.1
  [90137ffa] + StaticArrays v1.3.1
  [82ae8749] + StatsAPI v1.2.0
  [2913bbd2] + StatsBase v0.33.14
  [4c63d2b9] + StatsFuns v0.9.14
  [7792a7ef] + StrideArraysCore v0.2.9
  [c3572dad] + Sundials v4.9.1
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.6.1
  [8290d209] + ThreadingUtilities v0.4.7
  [a2a6695c] + TreeViews v0.3.0
  [d5829a12] + TriangularSolve v0.1.9
  [3a884ed6] + UnPack v1.0.2
  [3d5dd08c] + VectorizationBase v0.21.23
  [700de1a5] + ZygoteRules v0.2.2
  [e33a78d0] + Hwloc_jll v2.7.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [f50d1b31] + Rmath_jll v0.3.0+0
  [fb77eaff] + Sundials_jll v5.2.1+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [4536629a] + OpenBLAS_jll
  [05823500] + OpenLibm_jll
  [bea87d4a] + SuiteSparse_jll
  [83775a58] + Zlib_jll
  [8e850b90] + libblastrampoline_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
Precompiling project...
  94 dependencies successfully precompiled in 103 seconds

(@v1.7) pkg> 

[ngam]

~(Side note: we are not doing the deactivation correctly. Could you test it on your end? Essentially, the environment will carry over even if you deactivate the conda env and thus will appear in the system-wide julia as well, I logged out and logged back in --- this is a macos "server" I have at home)~ Never mind.

[ngam]

What's the difference between

  [3f19e933] + p7zip_jll

and

  [3f19e933] + p7zip_jll v16.2.1+1

Does the v16.2.1+1 signify that it is being installed and just just being linked? Interestingly (but not surprisingly) they both refer to 3f19e933 which tells me that, like julia, all these packages are pointing to exact deps... So if we solve one problem of the main julia, we are likely not doing much unless we are pointing to exactly the same versions as they do (e.g. libgit2?)... This is all becoming clearly pointless in my mind. We could just build the default julia without any further linking (we can just customize it so that it is isolated)

[ngam]

cc @isuruf: Building julia against conda-forge's utilities likely has no benefit, except all the pain of chasing deps. This examples above illustrates why.

[ngam]

Also

[f43a241f] + Downloads v1.6.0

🤯

[ngam]

In our 1.7.1 release, things are different. I don't really know how to interpret all of this...

(jlia) ngam@mp15:~$ julia              
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.1 (2021-12-22)
 _/ |\__'_|_|_|\__'_|  |  https://github.com/conda-forge/julia-feedstock
|__/                   |

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908* (2021-12-22 19:35 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, ivybridge)
Environment:
  JULIA_DEPOT_PATH_BACKUP = 
  JULIA_PROJECT_BACKUP = 
  JULIA_LOAD_PATH_BACKUP = 
  JULIA_DEPOT_PATH = /Users/ngam/.Mambaforge-MacOSX-x86_64/envs/jlia/share/julia:
  JULIA_PROJECT = @jlia
  JULIA_LOAD_PATH = @:@jlia:@stdlib
  JULIA_SSL_CA_ROOTS_PATH_BACKUP = 
  JULIA_SSL_CA_ROOTS_PATH = /Users/ngam/.Mambaforge-MacOSX-x86_64/envs/jlia/ssl/cacert.pem

julia> 

(@jlia) pkg> add Sundials
    Updating registry at `~/.Mambaforge-MacOSX-x86_64/envs/jlia/share/julia/registries/General.toml`
   Resolving package versions...
    Updating `~/.Mambaforge-MacOSX-x86_64/envs/jlia/share/julia/environments/jlia/Project.toml`
  [c3572dad] + Sundials v4.9.1
    Updating `~/.Mambaforge-MacOSX-x86_64/envs/jlia/share/julia/environments/jlia/Manifest.toml`
  [79e6a3ab] + Adapt v3.3.3
  [4fba245c] + ArrayInterface v3.2.2
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.1
  [fa961155] + CEnum v0.4.1
  [2a0fbf3d] + CPUSummary v0.1.6
  [d360d2e6] + ChainRulesCore v1.11.4
  [9e997f8a] + ChangesOfVariables v0.1.2
  [fb6a15b2] + CloseOpenIntervals v0.1.4
  [38540f10] + CommonSolve v0.2.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.41.0
  [187b0558] + ConstructionBase v1.3.0
  [754358af] + DEDataArrays v0.2.0
  [9a962f9c] + DataAPI v1.9.0
  [864edb3b] + DataStructures v0.18.11
  [e2d170a0] + DataValueInterfaces v1.0.0
  [b429d917] + DensityInterface v0.4.0
  [2b5f629d] + DiffEqBase v6.80.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.9.0
  [31c24e10] + Distributions v0.25.37
  [ffbed154] + DocStringExtensions v0.8.6
  [7034ab61] + FastBroadcast v0.1.12
  [1a297f60] + FillArrays v0.12.7
  [6a86dc24] + FiniteDiff v2.9.0
  [f6369f11] + ForwardDiff v0.10.24
  [069b7b12] + FunctionWrappers v1.1.2
  [3e5b6fbb] + HostCPUFeatures v0.1.5
  [0e44f5e4] + Hwloc v2.0.0
  [615f187c] + IfElse v0.1.1
  [3587e190] + InverseFunctions v0.1.2
  [92d709cd] + IrrationalConstants v0.1.1
  [42fd0dbc] + IterativeSolvers v0.9.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [2ee39098] + LabelledArrays v1.7.0
  [10f19ff3] + LayoutPointers v0.1.4
  [2ab3a3ac] + LogExpFunctions v0.3.6
  [bdcacae8] + LoopVectorization v0.12.101
  [1914dd2f] + MacroTools v0.5.9
  [d125e4d3] + ManualMemory v0.1.6
  [e1d29d7a] + Missings v1.0.2
  [46d2c3a1] + MuladdMacro v0.2.2
  [77ba4419] + NaNMath v0.3.6
  [8913a72c] + NonlinearSolve v0.3.13
  [6fe1bfb0] + OffsetArrays v1.10.8
  [bac558e1] + OrderedCollections v1.4.1
  [90014a1f] + PDMats v0.11.5
  [d96e819e] + Parameters v0.12.3
  [f517fe37] + Polyester v0.6.2
  [1d0040c9] + PolyesterWeave v0.1.2
  [d236fae5] + PreallocationTools v0.2.1
  [21216c6a] + Preferences v1.2.3
  [1fd47b50] + QuadGK v2.4.2
  [3cdcf5f2] + RecipesBase v1.2.1
  [731186ca] + RecursiveArrayTools v2.24.0
  [f2c3362d] + RecursiveFactorization v0.2.7
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.2.0
  [79098fc4] + Rmath v0.7.0
  [3cdde19b] + SIMDDualNumbers v0.1.0
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.28
  [0bca4576] + SciMLBase v1.25.0
  [efcf1570] + Setfield v0.8.1
  [a2af1166] + SortingAlgorithms v1.0.1
  [276daf66] + SpecialFunctions v2.0.0
  [aedffcd0] + Static v0.4.1
  [90137ffa] + StaticArrays v1.3.1
  [82ae8749] + StatsAPI v1.2.0
  [2913bbd2] + StatsBase v0.33.14
  [4c63d2b9] + StatsFuns v0.9.14
  [7792a7ef] + StrideArraysCore v0.2.9
  [c3572dad] + Sundials v4.9.1
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.6.1
  [8290d209] + ThreadingUtilities v0.4.7
  [a2a6695c] + TreeViews v0.3.0
  [d5829a12] + TriangularSolve v0.1.9
  [3a884ed6] + UnPack v1.0.2
  [3d5dd08c] + VectorizationBase v0.21.23
  [700de1a5] + ZygoteRules v0.2.2
  [e33a78d0] + Hwloc_jll v2.7.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [f50d1b31] + Rmath_jll v0.3.0+0
  [fb77eaff] + Sundials_jll v5.2.1+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [4536629a] + OpenBLAS_jll
  [05823500] + OpenLibm_jll
  [bea87d4a] + SuiteSparse_jll
  [83775a58] + Zlib_jll
  [8e850b90] + libblastrampoline_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
Precompiling project...
  94 dependencies successfully precompiled in 104 seconds

[ngam]

Okay, now I think that was just an issue for our 1.8.0 build.

In our 1.7.1 build, you can still see the linking. The weird thing is, I have no idea where julia puts other libraries (e.g. sundials) it seems they only exist in a julia-specific compiled version?

(jlia) ngam@mp15:~/.Mambaforge-MacOSX-x86_64/envs/jlia/lib/julia$ ls -l
total 519904
-rwxrwxr-x  2 ngam  staff   62213584 Jan  8 23:37 libLLVM.dylib
-rwxrwxr-x  2 ngam  staff      42436 Jan  8 23:37 libamd.2.4.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libamd.2.dylib -> libamd.2.4.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libamd.dylib -> libamd.2.4.6.dylib
-rwxrwxr-x  2 ngam  staff      49664 Jan  8 23:37 libatomic.1.dylib
lrwxr-xr-x  1 ngam  staff         17 Jan 10 15:36 libatomic.dylib -> libatomic.1.dylib
-rwxrwxr-x  2 ngam  staff    2686976 Jan  8 23:37 libblastrampoline.dylib
-rwxrwxr-x  2 ngam  staff      20968 Jan  8 23:37 libbtf.1.2.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libbtf.1.dylib -> libbtf.1.2.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libbtf.dylib -> libbtf.1.2.6.dylib
-rwxrwxr-x  2 ngam  staff      46584 Jan  8 23:37 libcamd.2.4.6.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libcamd.2.dylib -> libcamd.2.4.6.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libcamd.dylib -> libcamd.2.4.6.dylib
-rwxrwxr-x  2 ngam  staff      30844 Jan  8 23:37 libccalltest.dylib
-rwxrwxr-x  2 ngam  staff      59048 Jan  8 23:37 libccolamd.2.9.6.dylib
lrwxr-xr-x  1 ngam  staff         22 Jan 10 15:36 libccolamd.2.dylib -> libccolamd.2.9.6.dylib
lrwxr-xr-x  1 ngam  staff         22 Jan 10 15:36 libccolamd.dylib -> libccolamd.2.9.6.dylib
-rwxrwxr-x  2 ngam  staff    1094764 Jan  8 23:37 libcholmod.3.0.14.dylib
lrwxr-xr-x  1 ngam  staff         23 Jan 10 15:36 libcholmod.3.dylib -> libcholmod.3.0.14.dylib
lrwxr-xr-x  1 ngam  staff         23 Jan 10 15:36 libcholmod.dylib -> libcholmod.3.0.14.dylib
-rwxrwxr-x  2 ngam  staff      33872 Jan  8 23:37 libcolamd.2.9.6.dylib
lrwxr-xr-x  1 ngam  staff         21 Jan 10 15:36 libcolamd.2.dylib -> libcolamd.2.9.6.dylib
lrwxr-xr-x  1 ngam  staff         21 Jan 10 15:36 libcolamd.dylib -> libcolamd.2.9.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libcurl.dylib -> ../libcurl.4.dylib
-rwxrwxr-x  2 ngam  staff      27552 Jan  8 23:37 libdSFMT.dylib
-rwxrwxr-x  2 ngam  staff     140824 Jan  8 23:37 libgcc_s.1.dylib
-rwxrwxr-x  2 ngam  staff    3273552 Jan  8 23:37 libgfortran.5.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libgfortran.dylib -> libgfortran.5.dylib
-rwxrwxr-x  2 ngam  staff    1109520 Jan  8 23:37 libgit2.1.1.0.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libgit2.1.1.dylib -> libgit2.1.1.0.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libgit2.dylib -> libgit2.1.1.0.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libgmp.dylib -> ../libgmp.10.dylib
-rwxrwxr-x  2 ngam  staff     361456 Jan  8 23:37 libgomp.1.dylib
lrwxr-xr-x  1 ngam  staff         15 Jan 10 15:36 libgomp.dylib -> libgomp.1.dylib
-rwxrwxr-x  2 ngam  staff    3241516 Jan  8 23:37 libjulia-internal.1.7.dylib
lrwxr-xr-x  1 ngam  staff         27 Jan 10 15:36 libjulia-internal.1.dylib -> libjulia-internal.1.7.dylib
lrwxr-xr-x  1 ngam  staff         27 Jan 10 15:36 libjulia-internal.dylib -> libjulia-internal.1.7.dylib
-rwxrwxr-x  2 ngam  staff     215036 Jan  8 23:37 libklu.1.3.8.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libklu.1.dylib -> libklu.1.3.8.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libklu.dylib -> libklu.1.3.8.dylib
-rwxrwxr-x  2 ngam  staff      17376 Jan  8 23:37 libldl.2.2.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libldl.2.dylib -> libldl.2.2.6.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libldl.dylib -> libldl.2.2.6.dylib
-rwxrwxr-x  2 ngam  staff      22968 Jan  8 23:37 libllvmcalltest.dylib
-rwxrwxr-x  2 ngam  staff     565488 Jan  8 23:37 libmbedcrypto.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         26 Jan 10 15:36 libmbedcrypto.5.dylib -> libmbedcrypto.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         26 Jan 10 15:36 libmbedcrypto.dylib -> libmbedcrypto.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         23 Jan 10 15:36 libmbedtls.13.dylib -> libmbedtls.2.24.0.dylib
-rwxrwxr-x  2 ngam  staff     273168 Jan  8 23:37 libmbedtls.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         23 Jan 10 15:36 libmbedtls.dylib -> libmbedtls.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         24 Jan 10 15:36 libmbedx509.1.dylib -> libmbedx509.2.24.0.dylib
-rwxrwxr-x  2 ngam  staff     160016 Jan  8 23:37 libmbedx509.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         24 Jan 10 15:36 libmbedx509.dylib -> libmbedx509.2.24.0.dylib
lrwxr-xr-x  1 ngam  staff         18 Jan 10 15:36 libmpfr.dylib -> ../libmpfr.6.dylib
lrwxr-xr-x  1 ngam  staff         22 Jan 10 15:36 libnghttp2.dylib -> ../libnghttp2.14.dylib
lrwxr-xr-x  1 ngam  staff         32 Jan 10 15:36 libopenblas64_.dylib -> ../libopenblas64_p-r0.3.18.dylib
lrwxr-xr-x  1 ngam  staff         24 Jan 10 15:36 libopenlibm.dylib -> ../libopenlibm.3.0.dylib
lrwxr-xr-x  1 ngam  staff         26 Jan 10 15:36 libpcre2-8.dylib -> ../libpcre2-8.0.10.2.dylib
-rwxrwxr-x  2 ngam  staff     320624 Jan  8 23:37 libquadmath.0.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libquadmath.dylib -> libquadmath.0.dylib
-rwxrwxr-x  2 ngam  staff      72000 Jan  8 23:37 librbio.2.2.6.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 librbio.2.dylib -> librbio.2.2.6.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 librbio.dylib -> librbio.2.2.6.dylib
-rwxrwxr-x  2 ngam  staff     230560 Jan  8 23:37 libspqr.2.0.9.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libspqr.2.dylib -> libspqr.2.0.9.dylib
lrwxr-xr-x  1 ngam  staff         19 Jan 10 15:36 libspqr.dylib -> libspqr.2.0.9.dylib
lrwxr-xr-x  1 ngam  staff         22 Jan 10 15:36 libssh2.dylib -> ../libssh2.1.0.1.dylib
-rwxrwxr-x  2 ngam  staff      17408 Jan  8 23:37 libssp.0.dylib
lrwxr-xr-x  1 ngam  staff         14 Jan 10 15:36 libssp.dylib -> libssp.0.dylib
-rwxrwxr-x  2 ngam  staff    3703664 Jan  8 23:37 libstdc++.6.dylib
lrwxr-xr-x  1 ngam  staff         17 Jan 10 15:36 libstdc++.dylib -> libstdc++.6.dylib
-rwxrwxr-x  2 ngam  staff      13268 Jan  8 23:37 libsuitesparseconfig.5.10.1.dylib
lrwxr-xr-x  1 ngam  staff         33 Jan 10 15:36 libsuitesparseconfig.5.dylib -> libsuitesparseconfig.5.10.1.dylib
lrwxr-xr-x  1 ngam  staff         33 Jan 10 15:36 libsuitesparseconfig.dylib -> libsuitesparseconfig.5.10.1.dylib
-rwxrwxr-x  2 ngam  staff     907512 Jan  8 23:37 libumfpack.5.7.9.dylib
lrwxr-xr-x  1 ngam  staff         22 Jan 10 15:36 libumfpack.5.dylib -> libumfpack.5.7.9.dylib
lrwxr-xr-x  1 ngam  staff         22 Jan 10 15:36 libumfpack.dylib -> libumfpack.5.7.9.dylib
-rwxrwxr-x  2 ngam  staff     306304 Jan  8 23:37 libuv.2.dylib
lrwxr-xr-x  1 ngam  staff         13 Jan 10 15:36 libuv.dylib -> libuv.2.dylib
-rwxrwxr-x  2 ngam  staff  168687476 Jan  8 23:37 sys.dylib
drwxr-xr-x  3 ngam  staff         96 Jan 10 15:36 sys.dylib.dSYM

[ngam]

So some libraries are under artifacts ... in the home directory (~/.julia/artifacts/) --- is this artifacts folder shared across envs, including ones inside conda envs? Otherwise, I cannot find an artifiacts folder inside the conda_env/share/julia_env directory

edit: I found this in /conda_env/share/julia_env/artifacts but in one env, not in both envs I have...

[ngam]

conda's sundials==6:

(jdev) ngam@mp15:~/.Mambaforge-MacOSX-x86_64/envs/jdev$ ls lib/libsundials*
lib/libsundials_arkode.5.0.0.dylib
lib/libsundials_arkode.5.dylib
lib/libsundials_arkode.dylib
lib/libsundials_cvode.6.0.0.dylib
lib/libsundials_cvode.6.dylib
lib/libsundials_cvode.dylib
lib/libsundials_cvodes.6.0.0.dylib
lib/libsundials_cvodes.6.dylib
lib/libsundials_cvodes.dylib
lib/libsundials_generic.6.0.0.dylib
lib/libsundials_generic.6.dylib
lib/libsundials_generic.dylib
lib/libsundials_ida.6.0.0.dylib
lib/libsundials_ida.6.dylib
lib/libsundials_ida.dylib
lib/libsundials_idas.5.0.0.dylib
lib/libsundials_idas.5.dylib
lib/libsundials_idas.dylib
lib/libsundials_kinsol.6.0.0.dylib
lib/libsundials_kinsol.6.dylib
lib/libsundials_kinsol.dylib
lib/libsundials_nvecmanyvector.6.0.0.dylib
lib/libsundials_nvecmanyvector.6.dylib
lib/libsundials_nvecmanyvector.dylib
lib/libsundials_nvecserial.6.0.0.dylib
lib/libsundials_nvecserial.6.dylib
lib/libsundials_nvecserial.dylib
lib/libsundials_sunlinsolband.4.0.0.dylib
lib/libsundials_sunlinsolband.dylib
lib/libsundials_sunlinsoldense.4.0.0.dylib
lib/libsundials_sunlinsoldense.dylib
lib/libsundials_sunlinsolklu.4.0.0.dylib
lib/libsundials_sunlinsolklu.dylib
lib/libsundials_sunlinsollapackband.4.0.0.dylib
lib/libsundials_sunlinsollapackband.dylib
lib/libsundials_sunlinsollapackdense.4.0.0.dylib
lib/libsundials_sunlinsollapackdense.dylib
lib/libsundials_sunlinsolpcg.4.0.0.dylib
lib/libsundials_sunlinsolpcg.dylib
lib/libsundials_sunlinsolspbcgs.4.0.0.dylib
lib/libsundials_sunlinsolspbcgs.dylib
lib/libsundials_sunlinsolspfgmr.4.0.0.dylib
lib/libsundials_sunlinsolspfgmr.dylib
lib/libsundials_sunlinsolspgmr.4.0.0.dylib
lib/libsundials_sunlinsolspgmr.dylib
lib/libsundials_sunlinsolsptfqmr.4.0.0.dylib
lib/libsundials_sunlinsolsptfqmr.dylib
lib/libsundials_sunmatrixband.4.0.0.dylib
lib/libsundials_sunmatrixband.4.dylib
lib/libsundials_sunmatrixband.dylib
lib/libsundials_sunmatrixdense.4.0.0.dylib
lib/libsundials_sunmatrixdense.4.dylib
lib/libsundials_sunmatrixdense.dylib
lib/libsundials_sunmatrixsparse.4.0.0.dylib
lib/libsundials_sunmatrixsparse.4.dylib
lib/libsundials_sunmatrixsparse.dylib
lib/libsundials_sunnonlinsolfixedpoint.3.0.0.dylib
lib/libsundials_sunnonlinsolfixedpoint.dylib
lib/libsundials_sunnonlinsolnewton.3.0.0.dylib
lib/libsundials_sunnonlinsolnewton.dylib

julia's sundials:

(jdev) ngam@mp15:~/.Mambaforge-MacOSX-x86_64/envs/jdev$ ls share/julia/artifacts/*/lib/libsundials*
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_arkode.4.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_arkode.4.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_arkode.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_arkode.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvode.5.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvode.5.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvode.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvode.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvodes.5.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvodes.5.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvodes.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_cvodes.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_ida.5.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_ida.5.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_ida.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_ida.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_idas.4.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_idas.4.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_idas.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_idas.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_kinsol.5.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_kinsol.5.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_kinsol.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_kinsol.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecmanyvector.5.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecmanyvector.5.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecmanyvector.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecmanyvector.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecserial.5.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecserial.5.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecserial.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_nvecserial.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolband.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolband.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolband.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolband.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsoldense.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsoldense.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsoldense.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsoldense.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolklu.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolklu.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolklu.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolklu.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackband.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackband.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackband.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackband.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackdense.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackdense.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackdense.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsollapackdense.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolpcg.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolpcg.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolpcg.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolpcg.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspbcgs.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspbcgs.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspbcgs.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspbcgs.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspfgmr.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspfgmr.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspfgmr.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspfgmr.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspgmr.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspgmr.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspgmr.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolspgmr.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolsptfqmr.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolsptfqmr.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolsptfqmr.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunlinsolsptfqmr.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixband.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixband.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixband.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixband.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixdense.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixdense.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixdense.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixdense.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixsparse.3.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixsparse.3.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixsparse.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunmatrixsparse.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolfixedpoint.2.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolfixedpoint.2.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolfixedpoint.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolfixedpoint.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolnewton.2.2.0.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolnewton.2.dylib
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolnewton.a
share/julia/artifacts/c58c2773ab8fec9b371a170c79647619aa4fb7b4/lib/libsundials_sunnonlinsolnewton.dylib

---

conda's sundials==5:

(jdev) ngam@mp15:~/.Mambaforge-MacOSX-x86_64/envs/jdev$ ls lib/libsundials*                        
lib/libsundials_arkode.4.0.0.dylib
lib/libsundials_arkode.4.dylib
lib/libsundials_arkode.dylib
lib/libsundials_cvode.5.0.0.dylib
lib/libsundials_cvode.5.dylib
lib/libsundials_cvode.dylib
lib/libsundials_cvodes.5.0.0.dylib
lib/libsundials_cvodes.5.dylib
lib/libsundials_cvodes.dylib
lib/libsundials_ida.5.0.0.dylib
lib/libsundials_ida.5.dylib
lib/libsundials_ida.dylib
lib/libsundials_idas.4.0.0.dylib
lib/libsundials_idas.4.dylib
lib/libsundials_idas.dylib
lib/libsundials_kinsol.5.0.0.dylib
lib/libsundials_kinsol.5.dylib
lib/libsundials_kinsol.dylib
lib/libsundials_nvecmanyvector.5.0.0.dylib
lib/libsundials_nvecmanyvector.5.dylib
lib/libsundials_nvecmanyvector.dylib
lib/libsundials_nvecserial.5.0.0.dylib
lib/libsundials_nvecserial.5.dylib
lib/libsundials_nvecserial.dylib
lib/libsundials_sunlinsolband.3.0.0.dylib
lib/libsundials_sunlinsolband.3.dylib
lib/libsundials_sunlinsolband.dylib
lib/libsundials_sunlinsoldense.3.0.0.dylib
lib/libsundials_sunlinsoldense.3.dylib
lib/libsundials_sunlinsoldense.dylib
lib/libsundials_sunlinsolklu.3.0.0.dylib
lib/libsundials_sunlinsolklu.3.dylib
lib/libsundials_sunlinsolklu.dylib
lib/libsundials_sunlinsollapackband.3.0.0.dylib
lib/libsundials_sunlinsollapackband.3.dylib
lib/libsundials_sunlinsollapackband.dylib
lib/libsundials_sunlinsollapackdense.3.0.0.dylib
lib/libsundials_sunlinsollapackdense.3.dylib
lib/libsundials_sunlinsollapackdense.dylib
lib/libsundials_sunlinsolpcg.3.0.0.dylib
lib/libsundials_sunlinsolpcg.3.dylib
lib/libsundials_sunlinsolpcg.dylib
lib/libsundials_sunlinsolspbcgs.3.0.0.dylib
lib/libsundials_sunlinsolspbcgs.3.dylib
lib/libsundials_sunlinsolspbcgs.dylib
lib/libsundials_sunlinsolspfgmr.3.0.0.dylib
lib/libsundials_sunlinsolspfgmr.3.dylib
lib/libsundials_sunlinsolspfgmr.dylib
lib/libsundials_sunlinsolspgmr.3.0.0.dylib
lib/libsundials_sunlinsolspgmr.3.dylib
lib/libsundials_sunlinsolspgmr.dylib
lib/libsundials_sunlinsolsptfqmr.3.0.0.dylib
lib/libsundials_sunlinsolsptfqmr.3.dylib
lib/libsundials_sunlinsolsptfqmr.dylib
lib/libsundials_sunmatrixband.3.0.0.dylib
lib/libsundials_sunmatrixband.3.dylib
lib/libsundials_sunmatrixband.dylib
lib/libsundials_sunmatrixdense.3.0.0.dylib
lib/libsundials_sunmatrixdense.3.dylib
lib/libsundials_sunmatrixdense.dylib
lib/libsundials_sunmatrixsparse.3.0.0.dylib
lib/libsundials_sunmatrixsparse.3.dylib
lib/libsundials_sunmatrixsparse.dylib
lib/libsundials_sunnonlinsolfixedpoint.2.0.0.dylib
lib/libsundials_sunnonlinsolfixedpoint.2.dylib
lib/libsundials_sunnonlinsolfixedpoint.dylib
lib/libsundials_sunnonlinsolnewton.2.0.0.dylib
lib/libsundials_sunnonlinsolnewton.2.dylib
lib/libsundials_sunnonlinsolnewton.dylib

Can we expose these to each other? In our case, preferably, have julia just read from our lib folder. Or put differently, could we manipulate how this artifacts folder is constructed by julia? Resolving them correctly is a different matter, obvz

[mkitti]

Again, you need to provide an alternate version of Sundials_jll.jl. By default, it is pulling this one: https://github.com/JuliaBinaryWrappers/Sundials_jll.jl.

That Sundials_jll.jl specifies its artifacts here: https://github.com/JuliaBinaryWrappers/Sundials_jll.jl/blob/main/Artifacts.toml

julia> using Sundials_jll

julia> names(Sundials_jll)
24-element Vector{Symbol}:
 :Sundials_jll
 :libsundials_arkode
 :libsundials_cvode
 :libsundials_cvodes
 :libsundials_ida
 :libsundials_idas
 :libsundials_kinsol
 :libsundials_nvecmanyvector
 :libsundials_nvecserial
 :libsundials_sunlinsolband
 ⋮
 :libsundials_sunlinsolspbcgs
 :libsundials_sunlinsolspfgmr
 :libsundials_sunlinsolspgmr
 :libsundials_sunlinsolsptfqmr
 :libsundials_sunmatrixband
 :libsundials_sunmatrixdense
 :libsundials_sunmatrixsparse
 :libsundials_sunnonlinsolfixedpoint
 :libsundials_sunnonlinsolnewton

julia> Sundials_jll.libsundials_cvode
"/home/mkitti/anaconda3/envs/julia180dev1258/share/julia/artifacts/cb7fc2801ca0133a5bdea4bc4585d07c08284cfa/lib/libsundials_cvode.so"

julia> Sundials_jll.libsundials_cvode_path
"/home/mkitti/anaconda3/envs/julia180dev1258/share/julia/artifacts/cb7fc2801ca0133a5bdea4bc4585d07c08284cfa/lib/libsundials_cvode.so"

All the Julia package Sundials_jll does is provides a path for all the libraries needed by Sundials. With this default version of Sundials_jll, there are several ways to override the paths. One is to use https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artifact-locations. The other is to create a directory within the artifact called "overrides" and provide a bunch of symlinks there.

We could provide an alternate package, perhaps at https://github.com/conda-forge/Sundials_jll that would just have

module Sundials_jll

using Libdl

@static if Sys.isapple()
    const ext = ".dylib"
elseif Sys.islinux()
    const ext = ".so"
end

const libsundials_cvode_path = joinpath(ENV["CONDA_PREFIX"], "lib", "libsundials_cvode$ext")
const libsundials_cvode = libsundials_cvode_path 
libsundials_cvode_handle = dlopen(Sundials_jll.libsundials_cvodes_path)
...
end

Now if you installed https://github.com/conda-forge/Sundials_jll by doing ] add https://github.com/conda-forge/Sundials_jll then Sundials will use the binaries that points to. We would probably do that for the user if they typed conda -c conda-forge install Sundials_jll

A larger scale solution would be to create an alternate version of https://github.com/JuliaPackaging/JLLWrappers.jl where find_artifact_dir returns ENV["CONDA_PREFIX"].

[mkitti]

Review https://github.com/JuliaInterop/CxxWrap.jl/issues/309 and #164

[mkitti]

So some libraries are under artifacts ... in the home directory (~/.julia/artifacts/) --- is this artifacts folder shared across envs, including ones inside conda envs? Otherwise, I cannot find an artifiacts folder inside the conda_env/share/julia_env directory

edit: I found this in /conda_env/share/julia_env/artifacts but in one env, not in both envs I have...

Currently we have DEPOT_PATH as derived from JULIA_DEPOT_PATH setup as follows:

julia> DEPOT_PATH
4-element Vector{String}:
 "/home/mkitti/anaconda3/envs/julia180dev1258/share/julia"
 "/home/mkitti/.julia"
 "/home/mkitti/src/julia-1.7.1/local/share/julia"
 "/home/mkitti/src/julia-1.7.1/share/julia"

If you have anything in ~/.julia it might end up using that. If you don't want to use ~/.julia at all just delete that directory or move it somewhere. Another alternative is to remove the trailing colon from the environmental variable JULIA_DEPOT_PATH. A third session specific option is just do

julia> deleteat!(DEPOT_PATH, 2)
3-element Vector{String}:
 "/home/mkitti/anaconda3/envs/julia180dev1258/share/julia"
 "/home/mkitti/src/julia-1.7.1/local/share/julia"
 "/home/mkitti/src/julia-1.7.1/share/julia"

[mkitti]

What's the difference between

  [3f19e933] + p7zip_jll

and

  [3f19e933] + p7zip_jll v16.2.1+1

Does the v16.2.1+1 signify that it is being installed and just just being linked? Interestingly (but not surprisingly) they both refer to 3f19e933 which tells me that, like julia, all these packages are pointing to exact deps... So if we solve one problem of the main julia, we are likely not doing much unless we are pointing to exactly the same versions as they do (e.g. libgit2?)... This is all becoming clearly pointless in my mind. We could just build the default julia without any further linking (we can just customize it so that it is isolated)

The difference here is likely just between v1.7 and v1.8. I think v1.7 omits the version numbers for components that are part of the standard library. I believe both lines are actually doing the same thing, adding p7zip_jll into your julia environment . https://github.com/JuliaBinaryWrappers/p7zip_jll.jl. v1.7 just omitted the version from the verbose output for some reason.

3f19e933 is the first part of the UUID for p7zip_jll. Rather than relying on the package name, we give each package a UUID. https://github.com/JuliaBinaryWrappers/p7zip_jll.jl/blob/9a28d666b6acbc9ec9c3e006d6be4bfe5ce33dd8/Project.toml#L2

You can generate a UUID via:

julia> import UUIDs

julia> UUIDs.uuid4()
UUID("e1491eb0-ee47-4f16-9230-077ad728b523")

Within a Project.toml, there is also a [compat] section. For Sundials.jl, this is as follows:

[compat]
CEnum = "0.2, 0.3, 0.4"
DataStructures = "0.18"
DiffEqBase = "6.44"
Reexport = "0.2, 1.0"
Sundials_jll = "5.2"
julia = "1.6"

The interpretation of that is specified here: https://pkgdocs.julialang.org/v1/compatibility/

The above specifies the following compatability

• CEnum 0.2.*, 0.3.*, 0.4.*
• DataStructures 0.18.*
• DiffEqBase [6.44, 7)
• Reexport 0.2.x, [1.0, 2)
• julia [1.6, 2)

[mkitti]

• When is linking to system utilities (like we do) likely useful?

We're mostly just defining the versions of the libraries used in the standard library. If someone just writes Julia code, they will use the system binaries we specified.

If a package depends on a JLL or a package that has artifacts, then we need to indicate to Julia either through that JLL or via an alternate version of that JLL where the binaries that we want to use are.

[mkitti]

[f43a241f] + Downloads v1.6.0

This is not surprising. You start with a blank environment. When you start adding packages, then it will add the standard lib to your environment and then the exact versions of everything used are being added to the corresponding Manifest.toml.

The only thing that it happening here are two lines are added to Project.toml for your active Julia project:

(@julia180dev1258) pkg> add Downloads
[ Info: Use `./Downloads` to add or develop the local directory at `~/Downloads`.
    Updating registry at `~/anaconda3/envs/julia180dev1258/share/julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `~/anaconda3/envs/julia180dev1258/share/julia/environments/julia180dev1258/Project.toml`
  [f43a241f] + Downloads
    Updating `~/anaconda3/envs/julia180dev1258/share/julia/environments/julia180dev1258/Manifest.toml`
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [f43a241f] + Downloads
  [b27032c2] + LibCURL
  [8f399da3] + Libdl
  [ca575930] + NetworkOptions
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [83775a58] + Zlib_jll
  [8e850ede] + nghttp2_jll

julia> Base.active_project()
"/home/mkitti/anaconda3/envs/julia180dev1258/share/julia/environments/julia180dev1258/Project.toml"

julia> println.(readlines(Base.active_project()));
[deps]
Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

julia> println.(readlines(joinpath(dirname(Base.active_project()), "Manifest.toml") ));
# This file is machine-generated - editing it directly is not advised

julia_version = "1.7.1"
manifest_format = "2.0"

[[deps.ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[deps.Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[deps.Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[deps.LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[deps.LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[deps.LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[deps.Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[deps.MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[deps.MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[deps.NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[deps.Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[deps.nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[ngam]

@mkitti as always, thanks for patiently walking through these things! You're a star 🌟🌟🌟🌟🌟

I understand what you're saying about these options and I think those are great ideas. I am still trying to see if we can just have julia look into our $CONDA_PREFIX/* (lib/, include/, etc/, ...) before doing anything. I know that's dangerous (sometimes deps will be different versions, etc.) but we can likely have crude control measures in run_exports.

Basically, I think our "systemic" solution here may involve either overrides (as you mention) or some level of rewriting/editing the Pkg resolver (or whatever is happening underneath it). Or: manipulating how julia structures the directories, etc..

Anyway, ultimately here's what I think: There is no point in just having the standard library be from our system --- if that's all we can do, then we might as well have julia vendor these as well, I mean it is already doing its own thing in these subfolders and so on. This is, I think, is the ultimate problem here, and I don't know the solution. I don't want us to have to provide these julia packages again through conda --- instead, I just want julia to be aware* of the packages (all of them) already in our $CONDA_PREFIX (bin, lib, libexe, etc.). I read the entire Pkg documentation yesterday and I couldn't find any explanation of it resolves libraries (especially if something already exists). I just want it Pkg to scan some folders before deciding what to pull in; or artificially sym-link everything in $CONDA_PREFIX to the julia_prefix-equivalent.

*Note: same applies to conda --- I want conda to know what julia installs so that it doesn't reinstall it.

[mkitti]

If we look at Sundials_jll we see that the wrapper definition starts with a call to the macro @generate_wrapper_header. https://github.com/JuliaBinaryWrappers/Sundials_jll.jl/blob/ee3356bd471f3f7199b1232400eb460b9950b36b/src/wrappers/x86_64-linux-gnu-libgfortran5.jl#L7

JLLWrappers.@generate_wrapper_header("Sundials")

That macro is defined in JLLWrappers.jl here: https://github.com/JuliaPackaging/JLLWrappers.jl/blob/dd045e0576ddf834754100af4cc4829b236ae801/src/wrapper_generators.jl#L5-L19

macro generate_wrapper_header(src_name)
    pkg_dir = dirname(dirname(String(__source__.file)))
    return esc(quote
        function find_artifact_dir()
            # We determine at compile-time whether our JLL package has been dev'ed and overridden
            @static if isdir(joinpath(dirname($(pkg_dir)), "override"))
                return joinpath(dirname($(pkg_dir)), "override")
            else
                # We explicitly use `macrocall` here so that we can manually pass the `__source__`
                # argument, to avoid `@artifact_str` trying to lookup `Artifacts.toml` here.
                return $(Expr(:macrocall, Symbol("@artifact_str"), __source__, src_name))
            end
        end
    end)
end

Macros are basically just functions that operate on Julia expressions and that return modified Julia expressions to be evaluated. All this does is create a function find_artifact_dir() in Sundials_jll. If there is an override directory, then it will return that override directory. Otherwise, it will download and use the artifact. This override directory could be a symlink to $CONDA_PREFIX. The challenge then is when do you install this symlink?

Now if you wanted this to be more global, you could fork JLLWrappers.jl by doing

] dev JLLWrappers

and change the above macro to the following by editing $CONDA_PREFIX/share/julia/dev/JLLWrappers/src/wrapper_generators.jl

macro generate_wrapper_header(src_name)
    pkg_dir = dirname(dirname(String(__source__.file)))
    return esc(quote
        function find_artifact_dir()
            return ENV["CONDA_PREFIX"]
        end
    end)
end

Now all the JLL packages will point to $CONDA_PREFIX.

julia> using Sundials_jll

julia> Sundials_jll.find_artifact_dir()
"/home/mkitti/anaconda3/envs/julia180dev1258"

julia> ENV["CONDA_PREFIX"]
"/home/mkitti/anaconda3/envs/julia180dev1258"

julia> Sundials_jll.libsundials_cvode_path
"/home/mkitti/anaconda3/envs/julia180dev1258/lib/libsundials_cvode.so"

This might drive @giordano crazy, but as long as no one bothers him about it, it might be fine...

[ngam]

Can we ship a modified JLLWrappers with julia-cf?

[ngam]

Or we can have that be part of the activation/startup script?