Can't the JULIA_DEPOT_PATH be changed in apps?

[olivierlabayle]

Hello,

I am building an app in a docker container and then try to run it via singularity. The app builds successfully but can't run because the DEPOT_PATH seems to be hardlinked to the app's share folder. Whether I do:

./tmle/bin/TargetedEstimation --help

or

JULIA_DEPOT_PATH=/opt ./TargetedEstimation --help # or export JULIA_DEPOT_PATH=/opt

Scratch.jl which I think is supposed to write to the DEPOT_PATH seems to be tryiing to append the scratch_usage.toml in /TargetedEstimation.jl/tmle/share/julia/logs/.

Can I do anything to fix this? Thanks!

Full stacktrace:

fatal: error thrown and no exception handler available.
InitError(mod=:OpenML, error=Base.SystemError(prefix="opening file "/TargetedEstimation.jl/tmle/share/julia/logs/scratch_usage.toml"", errnum=30, extrainfo=nothing))
#systemerror#83 at ./error.jl:176
systemerror at ./error.jl:176
systemerror at ./error.jl:176
#systemerror#82 at ./error.jl:175 [inlined]
systemerror at ./error.jl:175 [inlined]
#open#705 at ./iostream.jl:293
open at ./iostream.jl:275 [inlined]
#open#409 at ./io.jl:393
open at ./io.jl:392 [inlined]
track_scratch_access at /opt/packages/Scratch/NVOrR/src/Scratch.jl:178
get_scratch! at /opt/packages/Scratch/NVOrR/src/Scratch.jl:240
macro expansion at /opt/packages/Scratch/NVOrR/src/Scratch.jl:306 [inlined]
__init__ at /opt/packages/OpenML/zbyTP/src/OpenML.jl:16
jfptr___init___114432 at /TargetedEstimation.jl/tmle/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_module_run_initializer at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:75
_finish_julia_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/init.c:855
julia_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/init.c:804
ijl_init_with_image at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:66 [inlined]
ijl_init_with_image at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:55
ijl_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:82
main at ./tmle/bin/TargetedEstimation (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at ./tmle/bin/TargetedEstimation (unknown line)

[kpamnany]

We've run into a similar problem. The hard-coded depot path (here) is not writeable in our environment. We want to dynamically create a package from the app so we need to add a writeable depot path. But since JULIA_DEPOT_PATH is ignored, we are unable to do this.

Can we change the setenv to instead do an append to whatever was already in JULIA_DEPOT_PATH?

Cc: @KristofferC

[kpamnany]

To retain current behavior, maybe appending to JULIA_DEPOT_PATH could be made conditional on a PackageCompiler command line argument?

[KristofferC]

You can change the environment on the app or? The one set there is only for starting it up. I must be misunderstanding something. But sure, we could append to it.

[kpamnany]

Yeah, we'd prefer to not hard-code environment changes into the app. Basically what happens now is we run:

$ export JULIA_DEPOT_PATH=...
$ <our_app>

And in <our_app>, we don't see what was set for JULIA_DEPOT_PATH (as it is overwritten by the setenv).