search

codedownio / julia2nix

Generate Nix expressions for building a Julia depot with a set of packages
53 stars 6 forks source link

julia precompiles Pluto although it is already compiled in the depot #22

Closed stuebinm closed 2 years ago

stuebinm commented 3 years ago

I have a Project.toml that just depends on the Pluto package. julia2nix generates a julia depot from it (with precompile set to true), and there is a compiled version of Pluto in it:

$: ll /nix/store/zqp2wz3426knbcqxm5j1jml74am8j84f-julia-depot/compiled/v1.5/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  Configurations/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  Crayons/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  ExproniconLite/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  FuzzyCompletions/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  HTTP/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  IniFile/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  JLLWrappers/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  MbedTLS/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  MbedTLS_jll/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  MsgPack/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  NetworkOptions/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  OrderedCollections/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  Pluto/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  TableIOInterface/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  TOML/
4 dr-xr-xr-x 2 root root 4096  1. Jan 1970  URIs/
$: ll /nix/store/zqp2wz3426knbcqxm5j1jml74am8j84f-julia-depot/compiled/v1.5/Pluto/
3040 -r--r--r-- 1 root root 3112076  1. Jan 1970  OJqMt_j0PSs.ji

However, the wrapped version of julia still precompiles Pluto (in ~/.julia) upon trying to use it:

$: result/bin/julia              _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.4 (2021-03-11)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using Pkg

julia> Pkg.activate(".")
 Activating environment at `~/juliatest/test2/Project.toml`

julia> using Pluto
[ Info: Precompiling Pluto [c3e4b0f8-55cb-11ea-2926-15256bba5781]
┌ Info: 
│ 
│     Welcome to Pluto v0.14.5 🎈
│     Start a notebook server using:
│ 
│   julia> Pluto.run()
│ 
│     Have a look at the FAQ:
│     https://github.com/fonsp/Pluto.jl/wiki
└

Interestingly, the filename of the compiled blob differs:

$: ll ~/.julia/compiled/v1.5/Pluto/
3052 -r--r--r-- 1 stuebinm users 3124939 20. Mai 10:34 OJqMt_1rmJN.ji

I used the same version of julia throughout (i.e. for building the package & manifest and in julia2nix). The uuid of the version precompiled in the repl is also the same as the one in the Project.toml (I'm not sure how to check for the already compiled version in the nix store, though).

However, I'm still pretty new to julia, so perhaps I just missed something obvious?

thomasjm commented 3 years ago

Keeping the precompile slugs consistent is kind of fragile unfortunately. It looks like your second one changed. The post here explains how it's calculated:

The part after the underscore is based on the project, the sysimage, the julia binary and something called “preferences”

We'll have to dig into exactly which of these is different when the depot is built vs. when you're using it. One way to go about it is to add print statements to Julia's base/loading.jl file in the compilecache_path function. Or to use other Julia debugging facilities; I'm not sure if those can be used without recompiling Julia. Want to try it?

stuebinm commented 3 years ago

So, after recompiling julia with a couple print() for debug output I think I've found at least a part of the problem: That second part of the hash contains the path to the currently used julia project:

While building via nix (components of the hash are in the same order in which they appear in compilecache_path:

this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
this is the base/loading.jl debug output: /build/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
┌ Info: 
│ 
│     Welcome to Pluto v0.14.5 🎈
│     Start a notebook server using:
│ 
│   julia> Pluto.run()
│ 
│     Have a look at the FAQ:
│     https://github.com/fonsp/Pluto.jl/wiki
└

Then later in the julia repl, when activating the environment and trying to use the precompiled packages (it fails since it only I only gave it access to the nix store here, but it does call compilecache_path once just before):

julia> using Pkg

julia> Pkg.activate(".")
[warning since I forced julia to only use the nix store depot, so it can't write logfiles]

julia> using Pluto
this is the base/loading.jl debug output: /home/stuebinm/juliatest-hack/Project.toml /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/bin/julia
[ Info: Precompiling Pluto [c3e4b0f8-55cb-11ea-2926-15256bba5781]
ERROR: SystemError: mktemp: Read-only file system
Stacktrace:
 [1] systemerror(::Symbol, ::Int32; extrainfo::Nothing) at /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so:?
 [2] (::Base.var"#systemerror##kw")(::NamedTuple{(:extrainfo,),Tuple{Nothing}}, ::typeof(systemerror), ::Symbol, ::Int32) at /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so:? (repeats 2 times)
 [3] compilecache(::Base.PkgId, ::String) at /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so:?
 [4] _require(::Base.PkgId) at /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so:?
 [5] require(::Base.PkgId) at /nix/store/qxrbzj4zh6vpiw6gamahaipar776cwcy-julia-1.5.4/lib/julia/sys.so:? (repeats 2 times)

During "normal" use without a Project.toml that path is something like ~/.julia/environments/v1.5/Project.toml, so packages can be re-used between sessions, but it appears that as soon as we use a Project.toml file this breaks down.

I'm not sure if there's a way around that, or if there is some option to suppress including the path here — tbh, I can't really see a reason why it is there (but then I'm fairly new to julia — does it have project-wide options that would change the compilation?)

What I did try was recompiling julia itself (though I cannot recommend that as a permanent solution, it takes ages), and hardcoding the second part of the slug to just "abcde", but this still fails and precompiles the packages again, although the filenames are exactly the same as in the depot in the nix store:

julia> using Pkg

julia> Pkg.activate(".")
 Activating environment at `~/juliatest-hack3/Project.toml`

julia> using Pluto
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/Pluto/OJqMt_abcde.ji
[ Info: Precompiling Pluto [c3e4b0f8-55cb-11ea-2926-15256bba5781]
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/Configurations/2z6N1_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/TOML/mjrwE_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/ExproniconLite/2CPrV_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/OrderedCollections/LtT3J_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/Crayons/TXPcU_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/FuzzyCompletions/xKHrE_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/MsgPack/oDgLV_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/HTTP/zXWya_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/URIs/eec2u_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/MbedTLS/bf9T0_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/MbedTLS_jll/u5NEn_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/JLLWrappers/7Zgw7_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/NetworkOptions/J8H6s_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/IniFile/wFzqC_abcde.ji
this is the base/loading.jl debug output:cachepath is: /home/stuebinm/.julia/compiled/v1.5/TableIOInterface/NPRph_abcde.ji
┌ Info: 
│ 
│     Welcome to Pluto v0.14.5 🎈
│     Start a notebook server using:
│ 
│   julia> Pluto.run()
│ 
│     Have a look at the FAQ:
│     https://github.com/fonsp/Pluto.jl/wiki
└ 
julia>

In the nix store e.g.

$: ls /nix/store/aib863c61v5djk1m2ii41jsgl20dxl1z-julia-depot/compiled/v1.5/Pluto/
OJqMt_abcde.ji

Even worse, it also fails when I try to manually copy the depot from the nix store to ~/.jula, make it writable, and tell it to use that. Looking at the content of the compiled blobs, they contain both the path to their source files and the Project.toml they were built from, so perhaps – for whatever reason they would be relevant – it also checks for those (see hexdump below)?

.jli.........Linux.x86_64.1.5.4.heads/v1.5.4.69fcb5745bda8a5588c089f7b65831787cffc366...
..Pluto...U.....W.k%.&)B'.4m"..............Y.../nix/store/aib863c61v5djk1m2ii41jsgl20dxl
1z-julia-depot/packages/Pluto/yDJcG/src/Pluto.jl..Qwa,.A...........i...r..R..Z..DPkg....
............Y.../nix/store/aib863c61v5djk1m2ii41jsgl20dxl1z-julia-depot/packages/Pluto/y
DJcG/Project.toml.}Qwa,.A....h.../nix/store/aib863c61v5djk1m2ii41jsgl20dxl1z-julia-depot
/packages/Pluto/yDJcG/src/notebook/PathHelpers.jlE.Rwa,.A........c.../nix/store/aib863c6
1v5djk1m2ii41jsgl20dxl1z-julia-depot/packages/Pluto/yDJcG/src/notebook/Export.jlE.Rwa,.A
............._........U.l.D.*Base64................a.../nix/store/aib863c61v5djk1m2ii41j

Anyways, I'm out of ideas for now, though I haven't quite given up one this yet.

thomasjm commented 3 years ago

Thanks for all the legwork so far!

I just remembered a trick I used before to debug this more easily. If you set the JULIA_DEBUG=loading environment variable as mentioned here, Julia outputs helpful information about what it's choosing to do with precompiled files.

I'm currently playing with the idea of removing the Base.active_project() part from the CRC, using the command below:

JULIA_DEBUG=loading $(nix-build . --no-out-link)/bin/julia -e 'using Pkg; Pkg.activate("."); using Pluto'
thomasjm commented 3 years ago

One thing I just noticed is that Julia seems to bake the mtime into the header of a .ji file at the time it's precompiled, which it later checks and uses to reject the cache file...

thomasjm commented 2 years ago

I just tried this with the latest version and was unable to reproduce the problem.

I'm going to assume that upstream Julia has improved the precompile fragility situation and close this, but please feel free to reopen if you see it again.