try run dynamic artifact selector in Pkg process

[KristofferC]

I don't really know what you are allowed to do in the dynamic artifact selector but this improves performance a lot at least (e.g. for Pkg.status) when packages using this are in the environment.

[giordano]

CC @vchuravy

[KristofferC]

WARNING: Method definition augment_platform!(Base.BinaryPlatforms.Platform) in module Operations at /tmp/jl_OfJNuL/AugmentedPlatform/.pkg/platform_augmentation.jl:4 overwritten at /tmp/jl_oUdeKZ/AugmentedPlatform/.pkg/platform_augmentation.jl:4.

Probably needs to be evaluated in their own module.

[KristofferC]

Something has to be changed here!! I added a @time before the select_cmd command begin run and do a Pkg.resolve

julia>  Pkg.resolve()
  1.018608 seconds (820 allocations: 5.425 MiB)
  0.680233 seconds (199 allocations: 1.072 MiB, 7.64% gc time)
  0.593827 seconds (406 allocations: 2.523 MiB)
  0.965070 seconds (1.05 k allocations: 7.056 MiB)
  0.635199 seconds (622 allocations: 4.038 MiB)
  0.597117 seconds (964 allocations: 6.426 MiB)
  0.553596 seconds (54 allocations: 65.938 KiB)
  0.627717 seconds (955 allocations: 6.366 MiB)
  0.566144 seconds (54 allocations: 65.938 KiB)
  No Changes to `~/JuliaPkgs/OmniPackage.jl/Project.toml`
  No Changes to `~/JuliaPkgs/OmniPackage.jl/Manifest.toml`
  1.004013 seconds (1.32 k allocations: 8.941 MiB)
  0.605631 seconds (1.27 k allocations: 8.561 MiB, 1.11% gc time)
  0.623127 seconds (1.58 k allocations: 10.707 MiB)
  1.005323 seconds (1.53 k allocations: 10.394 MiB)
  0.674921 seconds (1.09 k allocations: 7.313 MiB, 0.61% gc time)
  0.578730 seconds (577 allocations: 3.722 MiB)
  0.585688 seconds (54 allocations: 65.938 KiB)
  0.628440 seconds (919 allocations: 6.121 MiB)
  0.565631 seconds (54 allocations: 65.938 KiB)

Completely unacceptable.

[IanButterworth]

Alternatively we could try running this process without pkgimages?

Update:

I tried this. Perhaps not fast enough.

(@v1.12) pkg> dev https://github.com/JuliaComputing/OmniPackage.jl
     Cloning git-repo `https://github.com/JuliaComputing/OmniPackage.jl`
   Resolving package versions...
  1.871987 seconds (95 allocations: 67.562 KiB, 0.12% compilation time: 100% of which was recompilation)
  0.292939 seconds (54 allocations: 65.938 KiB)
  0.331011 seconds (1.39 k allocations: 9.374 MiB, 2.22% gc time)
  0.294084 seconds (54 allocations: 65.938 KiB)
  0.319483 seconds (54 allocations: 65.938 KiB)
  0.372198 seconds (811 allocations: 5.344 MiB, 16.00% gc time)
  0.294497 seconds (54 allocations: 65.938 KiB)
  0.293367 seconds (54 allocations: 65.938 KiB)

[KristofferC]

This is not so good (but can be trivially fixed):

julia> LOAD_PATH
57-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/kc/.julia/packages/CUDA_Runtime_jll/bmudn"
 "/home/kc/.julia/packages/P4est_jll/kfjxs"
 "/home/kc/.julia/packages/t8code_jll/RBsof"
 "/home/kc/.julia/packages/XGBoost_jll/WE2Ec"
 "/home/kc/.julia/packages/HDF5_jll/pjOXH"
 "/home/kc/.julia/packages/MPICH_jll/NeFHN"
 "/home/kc/.julia/packages/OpenMPI_jll/sHrC3"
 "/home/kc/.julia/packages/LLVMExtra_jll/iFERN"
 "/home/kc/.julia/packages/MPItrampoline_jll/ftT1U"
 "/home/kc/.julia/packages/CUDA_Runtime_jll/bmudn"
 "/home/kc/.julia/packages/P4est_jll/kfjxs"
 "/home/kc/.julia/packages/t8code_jll/RBsof"
 "/home/kc/.julia/packages/XGBoost_jll/WE2Ec"
 "/home/kc/.julia/packages/HDF5_jll/pjOXH"
 "/home/kc/.julia/packages/MPICH_jll/NeFHN"

Also packages seem to load the jll in their script:

https://github.com/JuliaBinaryWrappers/CUDA_Runtime_jll.jl/blob/6fc9e752fc55edbe606d8dd3a66b65469c2fbffa/.pkg/platform_augmentation.jl#L95-L99

so this would load the jll in the current process.

Sigh, this is all kinds of bad.

[IanButterworth]

Parallelize the artifact collection above this level?

[KristofferC]

I'm just sad to see us compiling virtually the same code over and over again in new processes just to do a stauts call even if that is running in parallel.

Maybe disable the is_instantiated in the status check for jlls using this functionality? Do some caching. I dunno...

[vchuravy]

Yeah we really want to execute this code in a sandbox and after running it throw away all side-effects.

But code-loading is currently hard to isolate... If we could unload modules that would be lovely...