PackageCompiler app can't install MKL on MS Windows 10

[bjarthur]

the problem has something to do with 7z.

here's the contents of testmkl/src/testmkl.jl:

module testmkl

using MKL

greet() = print("Hello World!")

#=
using PackageCompiler
create_app("./testmkl", "./testmklcompiled")
=#
function julia_main()::Cint
    println(MKL.INTERFACE_GNU)
    return 0
end

end # module testmkl

here's the error on the powershell command line when i try to run the app:

PS C:\Users\arthurb> .\testmklcompiled\bin\testmkl.exe
  Downloaded artifact: MKL
  Downloaded artifact: MKL
fatal: error thrown and no exception handler available.
InitError(mod=:MKL_jll, error=ErrorException("Unable to automatically download/install artifact 'MKL' from sources listed in 'C:\Users\arthurb\.julia\packages\MKL_jll\8Hu7G\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/6288063f1c2040dd23c7710d999838037ec279a5
    Error: IOError: could not spawn setenv(`7z.exe x 'C:\Users\arthurb\AppData\Local\Temp\jl_YjKR6haFGH-download.gz' -so`,["WINDIR=C:\\WINDOWS", "PATH=C:\\Users\\arthurb\\testmklcompiled\\bin\\..\\lib\\julia;C:\\Users\\arthurb\\testmklcompiled\\bin\\..\\lib;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Users\\arthurb\\AppData\\Local\\Microsoft\\WindowsApps;", "USERDOMAIN_ROAMINGPROFILE=HHMI", "LOCALAPPDATA=C:\\Users\\arthurb\\AppData\\Local", "HOMEPATH=\\", "PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 85 Stepping 4, GenuineIntel", "NUMBER_OF_PROCESSORS=32", "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL", "SESSIONNAME=RDP-Tcp#4", "SYSTEMROOT=C:\\WINDOWS", "APPDATA=C:\\Users\\arthurb\\AppData\\Roaming", "=::=::\\", "PSMODULEPATH=C:\\Users\\arthurb\\Documents\\WindowsPowerShell\\Modules;C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules", "COMMONPROGRAMW6432=C:\\Program Files\\Common Files", "PROGRAMDATA=C:\\ProgramData", "PUBLIC=C:\\Users\\Public", "USERDOMAIN=HHMI", "CLIENTNAME=arthurb-lm5", "OS=Windows_NT", "PROCESSOR_REVISION=5504", "TMP=C:\\Users\\arthurb\\AppData\\Local\\Temp", "COMMONPROGRAMFILES(X86)=C:\\Program Files (x86)\\Common Files", "COMSPEC=C:\\WINDOWS\\system32\\cmd.exe", "OPENBLAS_DEFAULT_NUM_THREADS=1", "UATDATA=C:\\WINDOWS\\CCM\\UATData\\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77", "ALLUSERSPROFILE=C:\\ProgramData", "COMMONPROGRAMFILES=C:\\Program Files\\Common Files", "COMPUTERNAME=ARTHURB-WW3", "JULIA_DEPOT_PATH=C:\\Users\\arthurb\\testmklcompiled\\share\\julia", "USERNAME=arthurb", "PROGRAMFILES(X86)=C:\\Program Files (x86)", "PROGRAMFILES=C:\\Program Files", "LOGONSERVER=\\\\JFDC3", "USERPROFILE=C:\\Users\\arthurb", "DRIVERDATA=C:\\Windows\\System32\\Drivers\\DriverData", "HOMESHARE=\\\\prfs.hhmi.org\\scicompsoft\$\\arthurb", "FPS_BROWSER_USER_PROFILE_STRING=Default", "JULIA_LOAD_PATH=C:\\Users\\arthurb\\testmklcompiled\\share\\julia", "PROCESSOR_LEVEL=6", "SYSTEMDRIVE=C:", "USERDNSDOMAIN=HHMI.ORG", "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer", "PROGRAMW6432=C:\\Program Files", "TEMP=C:\\Users\\arthurb\\AppData\\Local\\Temp", "HOMEDRIVE=U:", "OPENBLAS_MAIN_FREE=1", "PROCESSOR_ARCHITECTURE=AMD64"]): no such file or directory (ENOENT)
- https://github.com/JuliaBinaryWrappers/MKL_jll.jl/releases/download/MKL-v2023.2.0+0/MKL.v2023.2.0.x86_64-w64-mingw32.tar.gz
    Error: IOError: could not spawn setenv(`7z.exe x 'C:\Users\arthurb\AppData\Local\Temp\jl_mpypCP25EN-download.gz' -so`,["WINDIR=C:\\WINDOWS", "PATH=C:\\Users\\arthurb\\testmklcompiled\\bin\\..\\lib\\julia;C:\\Users\\arthurb\\testmklcompiled\\bin\\..\\lib;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Users\\arthurb\\AppData\\Local\\Microsoft\\WindowsApps;", "USERDOMAIN_ROAMINGPROFILE=HHMI", "LOCALAPPDATA=C:\\Users\\arthurb\\AppData\\Local", "HOMEPATH=\\", "PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 85 Stepping 4, GenuineIntel", "NUMBER_OF_PROCESSORS=32", "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL", "SESSIONNAME=RDP-Tcp#4", "SYSTEMROOT=C:\\WINDOWS", "APPDATA=C:\\Users\\arthurb\\AppData\\Roaming", "=::=::\\", "PSMODULEPATH=C:\\Users\\arthurb\\Documents\\WindowsPowerShell\\Modules;C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules", "COMMONPROGRAMW6432=C:\\Program Files\\Common Files", "PROGRAMDATA=C:\\ProgramData", "PUBLIC=C:\\Users\\Public", "USERDOMAIN=HHMI", "CLIENTNAME=arthurb-lm5", "OS=Windows_NT", "PROCESSOR_REVISION=5504", "TMP=C:\\Users\\arthurb\\AppData\\Local\\Temp", "COMMONPROGRAMFILES(X86)=C:\\Program Files (x86)\\Common Files", "COMSPEC=C:\\WINDOWS\\system32\\cmd.exe", "OPENBLAS_DEFAULT_NUM_THREADS=1", "UATDATA=C:\\WINDOWS\\CCM\\UATData\\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77", "ALLUSERSPROFILE=C:\\ProgramData", "COMMONPROGRAMFILES=C:\\Program Files\\Common Files", "COMPUTERNAME=ARTHURB-WW3", "JULIA_DEPOT_PATH=C:\\Users\\arthurb\\testmklcompiled\\share\\julia", "USERNAME=arthurb", "PROGRAMFILES(X86)=C:\\Program Files (x86)", "PROGRAMFILES=C:\\Program Files", "LOGONSERVER=\\\\JFDC3", "USERPROFILE=C:\\Users\\arthurb", "DRIVERDATA=C:\\Windows\\System32\\Drivers\\DriverData", "HOMESHARE=\\\\prfs.hhmi.org\\scicompsoft\$\\arthurb", "FPS_BROWSER_USER_PROFILE_STRING=Default", "JULIA_LOAD_PATH=C:\\Users\\arthurb\\testmklcompiled\\share\\julia", "PROCESSOR_LEVEL=6", "SYSTEMDRIVE=C:", "USERDNSDOMAIN=HHMI.ORG", "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer", "PROGRAMW6432=C:\\Program Files", "TEMP=C:\\Users\\arthurb\\AppData\\Local\\Temp", "HOMEDRIVE=U:", "OPENBLAS_MAIN_FREE=1", "PROCESSOR_ARCHITECTURE=AMD64"]): no such file or directory (ENOENT)
"))
error at .\error.jl:35
jfptr_error_32825.clone_1 at C:\Users\arthurb\testmklcompiled\lib\julia\sys.dll (unknown line)
#ensure_artifact_installed#23 at C:\Users\arthurb\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\share\julia\stdlib\v1.9\Pkg\src\Artifacts.jl:443
ensure_artifact_installed at C:\Users\arthurb\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\share\julia\stdlib\v1.9\Pkg\src\Artifacts.jl:387
unknown function (ip: 00000000622a7346)
#ensure_artifact_installed#22 at C:\Users\arthurb\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\share\julia\stdlib\v1.9\Pkg\src\Artifacts.jl:383
unknown function (ip: 000000006229cd16)
ensure_artifact_installed at C:\Users\arthurb\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\share\julia\stdlib\v1.9\Pkg\src\Artifacts.jl:372
unknown function (ip: 000000006229ca92)
_artifact_str at C:\Users\arthurb\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\share\julia\stdlib\v1.9\Artifacts\src\Artifacts.jl:551
unknown function (ip: 00000000622969f4)
jl_apply at C:/workdir/src\julia.h:1880 [inlined]
jl_f__call_latest at C:/workdir/src\builtins.c:774
#invokelatest#2 at .\essentials.jl:819 [inlined]
invokelatest at .\essentials.jl:816 [inlined]
macro expansion at C:\Users\arthurb\.julia\juliaup\julia-1.9.3+0.x64.w64.mingw32\share\julia\stdlib\v1.9\Artifacts\src\Artifacts.jl:703 [inlined]
find_artifact_dir at C:\Users\arthurb\.julia\packages\JLLWrappers\pG9bm\src\wrapper_generators.jl:17 [inlined]
__init__ at C:\Users\arthurb\.julia\packages\MKL_jll\8Hu7G\src\wrappers\x86_64-w64-mingw32.jl:9
jfptr___init___39249.clone_1 at C:\Users\arthurb\testmklcompiled\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1880 [inlined]
jl_module_run_initializer at C:/workdir/src\toplevel.c:75
_finish_julia_init at C:/workdir/src\init.c:855
ijl_init_with_image at C:/workdir/src\jlapi.c:66 [inlined]
ijl_init_with_image at C:/workdir/src\jlapi.c:55 [inlined]
ijl_init at C:/workdir/src\jlapi.c:82
.text at C:\Users\arthurb\testmklcompiled\bin\testmkl.exe (unknown line)
__tmainCRTStartup at C:\Users\arthurb\testmklcompiled\bin\testmkl.exe (unknown line)
.l_start at C:\Users\arthurb\testmklcompiled\bin\testmkl.exe (unknown line)
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)

[ViralBShah]

Maybe it can't find 7z? I wonder if this is a PackageCompiler issue?

cc @KristofferC

[bjarthur]

in case it matters:

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 32 × Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
  Threads: 1 on 32 virtual cores

[bjarthur]

for what it's worth, i tried following https://github.com/JuliaLinearAlgebra/MKL.jl/pull/20#issuecomment-565245124, and couldn't find 7z, despite being able to find it in the search box on windows, but i'm guessing MKL is trying to find the one in p7zip_jll.

julia> searchpath = ENV["PATH"]
"C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\Git\\cmd;C:\\Users\\arthurb\\AppData\\Local\\Microsoft\\WindowsApps;"

julia> if isdefined(Base, :LIBEXECDIR)
           sepchar = Sys.iswindows() ? ";" : ":"
           searchpath = string(joinpath(Sys.BINDIR, Base.LIBEXECDIR), sepchar, searchpath)
       end
"C:\\Users\\arthurb\\.julia\\juliaup\\julia-1.9.3+0.x64.w64.mingw32\\bin\\..\\libexec;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\Git\\cmd;C:\\Users\\arthurb\\AppData\\Local\\Microsoft\\WindowsApps;"

julia> sevenz_path = withenv("PATH" => searchpath) do
           Sys.which("7z")
       end

julia> sevenz_path = withenv("PATH" => searchpath) do
           Sys.which("7z.exe")
       end

[KristofferC]

https://github.com/JuliaLang/PackageCompiler.jl/pull/877 should fix this.

[bjarthur]

thanks @KristofferC . i'm testing that PR now, as requested.

i guess the bigger picture is that i don't understand why MKL is downloaded at runtime, instead of being bundled into the app when it is compiled.

[ViralBShah]

I believe that is how artifacts work - in that the right architecture specific version gets picked up at runtime. There may/should be a way to package it all by picking defaults at build time.

[bjarthur]

fixed by https://github.com/JuliaLang/PackageCompiler.jl/pull/877