Shell mode doesn't respect Cmd object's env

[ericphanson]

Taking this example from the docs:

julia> cmd = Cmd(`sh -c "echo foo \$HOWLONG"`, env=("HOWLONG" => "ever!",));

julia> run(cmd);
foo ever!

shell> $cmd
foo

using

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 4 on 4 virtual cores
Environment:
  JULIA_NUM_THREADS = 4
  JULIA_PKG_SERVER_REGISTRY_PREFERENCE = eager

One reason this is an issue is if you use JLL executables with shell mode, it doesn't have the paths setup correctly. E.g.

julia> run(`$(tectonic()) summary.tex`); # works
Running TeX ...
Rerunning TeX because "summary.aux" changed ...
Running xdvipdfmx ...
Writing `summary.pdf` (7.59 KiB)
Skipped writing 1 intermediate files (use --keep-intermediates to keep them)

shell> $(tectonic()) summary.tex
dyld[83200]: Library not loaded: @rpath/libharfbuzz-icu.0.dylib
  Referenced from: <A2B70B51-60AB-39C3-B62A-D51CC21629C0> /Users/eph/.julia/artifacts/26fe46dec594cb9f068729889006a2ec139f2511/bin/tectonic
  Reason: tried: '/Users/eph/.julia/artifacts/26fe46dec594cb9f068729889006a2ec139f2511/bin/../lib/libharfbuzz-icu.0.dylib' (no such file), '/Users/eph/.julia/artifacts/26fe46dec594cb9f068729889006a2ec139f2511/bin/../lib/libharfbuzz-icu.0.dylib' (no such file)

using tectonic_jll.

[Keno]

Yeah, this should definitely work.

[Keno]

Same semantics as regular cmd interpolation. The env modification applies if you're interpolating at the beginning of the cmd.

[ericphanson]

I started looking into this. I think we can improve things by changing this line

https://github.com/JuliaLang/julia/blob/0c46852901c63b33f0603b0afd58ff1da687d760/base/client.jl#L68

to something like

 cmd = Cmd(`$shell -c $shell_escape_cmd`, cmd.ignorestatus, cmd.flags, cmd.env, cmd.dir)

to correctly inherit the env and other flags.

However, that doesn't seem to be enough on macos to run JLLs from the shell mode. I think it could be MacOS SIP somehow interfering. If I do

run(Cmd(`bash -c 'printenv BLARG'`; env=["BLARG=hi"]))

then I get hi printed as expected. But if I do say

run(Cmd(`bash -c 'printenv DYLD_FALLBACK_LIBRARY_PATH'`;env=["DYLD_FALLBACK_LIBRARY_PATH=/Users/eph/.julia/artifacts/c5782c1d2e847459c5f931ce12fb1876015c37b7/lib:/Users/eph/.julia/artifacts/2a86ef020f132332b2f4be2fb40912cc7df5da29/lib:/Users/eph/julia-native/usr/lib:/Users/eph/.julia/artifacts/c65e07e3da4f1bf519bc432389dbbd61df320457/lib:/Users/eph/.julia/artifacts/7f4d1479db8bfb628aff3806c483e5fec617271a/lib:/Users/eph/julia-native/usr/bin/../lib/julia:/Users/eph/julia-native/usr/bin/../lib:/Users/eph/lib:/usr/local/lib:/lib:/usr/lib"]))

then I get an error, the same as if there was no env named DYLD_FALLBACK_LIBRARY_PATH. I believe this is coming into play here, where the command we are executing is something like bash -c ...run jll... and even if we set the correct env (with DYLD_FALLBACK_LIBRARY_PATH), it is getting stripped out by SIP before it executes and the JLL doesn't see the right DYLD_FALLBACK_LIBRARY_PATH.

One question I have is why are we actually shelling out (as opposed to just running the commands)? I know it is "shell" mode, but we don't support globbing or anything like that, so it doesn't seem that useful?