Per chat with @omus, there is a much better way to specify which dependencies to include in the sysimage that does not involve tight_deps.sh-like ugliness:
Use Pkg.jl's package pinning mechanism instead. Package pinning shows up in the Manifest.toml and is accessible from Pkg.dependencies() values as an is_pinned flag.
From @omus:
The current logic used to determine what packages should be included in the system image uses a file called TightProject.toml which uses the = compat operator to specify an exact version of package. The logic used to do this is rather frail at the moment and can break. Since Julia already uses Manifest.toml to keep track of an exact list of packages we should make use of that infrastructure to make julia_pod more reliable and ergonomic.
An outline of the proposal:
Use Julia's pinned package mechanism to allow julia_pod users to specify what packages should be included in the system image. This information is recorded in the project's Manifest.toml file.
By using pinned packages julia_pod users will be unable to accidentally change the package version of something baked into the system image. This provides a nice barrier which avoids confusion between the version reported by Pkg and the version of a package in the system image.
julia_pod will read the project Manifest.toml to find all the pinned packages as well their transitive dependencies. From this we'll generate a SysimgManifest.toml and SysimageProject.toml
In the Dockerfile the Sysimg*.toml files will be COPY'ed. Afterwards we'll instantiate the packages and build a Julia sysimg which includes all of those packages. The next step will COPY the Manifest.toml and Project.toml into the Docker container and then instantiate the packages. This should allow for system images to only be re-built when the list of pinned packages changed or the associated versions.
Optional: We could make use of Julia's stacked package environments to have both of the manifest files co-exist in a Julia aware way (e.g. JULIA_LOAD_PATH="/project:/sysimg:@v#.#:@stdlib"). Mainly this is just a convenience to be able to view what packages were included in the system image.
Example of package environment stacking working. I wanted to verify if a pinned package lower down the stack influenced a package higher on the stack.
```
❯ JULIA_LOAD_PATH="@stdlib" julia -e 'using Pkg; Pkg.activate("/tmp/sysimg"); Pkg.add(PackageSpec(name="Example", version="0.5.0")); Pkg.pin("Example"); Pkg.status()'
Activating new environment at `/tmp/sysimg/Project.toml`
Updating registry at `~/.julia/registries/Beacon`
Updating git-repo `https://github.com/beacon-biosignals/BeaconRegistry.git`
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Updating `/private/tmp/sysimg/Project.toml`
[7876af07] + Example v0.5.0
Updating `/private/tmp/sysimg/Manifest.toml`
[7876af07] + Example v0.5.0
0 dependencies successfully precompiled in 0 seconds
1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
Resolving package versions...
Updating `/private/tmp/sysimg/Project.toml`
[7876af07] ~ Example v0.5.0 ⇒ v0.5.0 ⚲
Updating `/private/tmp/sysimg/Manifest.toml`
[7876af07] ~ Example v0.5.0 ⇒ v0.5.0 ⚲
0 dependencies successfully precompiled in 0 seconds
1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
Status `/private/tmp/sysimg/Project.toml`
[7876af07] Example v0.5.0 ⚲
❯ JULIA_LOAD_PATH="/tmp/sysimg:@stdlib" julia -e 'using Pkg; Pkg.status()'
Status `/private/tmp/sysimg/Project.toml`
[7876af07] Example v0.5.0 ⚲
❯ JULIA_LOAD_PATH="/tmp/sysimg:@v#.#:@stdlib" julia -e 'using Pkg; Pkg.activate("/tmp/user"); Pkg.add(PackageSpec(name="Example", version="0.5.3")); Pkg.status()'
Activating new environment at `/tmp/user/Project.toml`
Updating registry at `~/.julia/registries/Beacon`
Updating git-repo `https://github.com/beacon-biosignals/BeaconRegistry.git`
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Installed Example ─ v0.5.3
Updating `/private/tmp/user/Project.toml`
[7876af07] + Example v0.5.3
Updating `/private/tmp/user/Manifest.toml`
[7876af07] + Example v0.5.3
Status `/private/tmp/user/Project.toml`
[7876af07] Example v0.5.3
❯ cat /tmp/sysimg/Manifest.toml
# This file is machine-generated - editing it directly is not advised
[[Example]]
git-tree-sha1 = "29a04c9d4de9e15249e84549d6f0eb107675c639"
pinned = true
uuid = "7876af07-990d-54b4-ab0e-23690620f79a"
version = "0.5.0"
❯ cat /tmp/user/Manifest.toml
# This file is machine-generated - editing it directly is not advised
[[Example]]
git-tree-sha1 = "46e44e869b4d90b96bd8ed1fdcf32244fddfb6cc"
uuid = "7876af07-990d-54b4-ab0e-23690620f79a"
version = "0.5.3"
```
kolia
commented
3 years ago
Per chat with @omus, there is a much better way to specify which dependencies to include in the sysimage that does not involve tight_deps.sh-like ugliness:
Use Pkg.jl's package pinning mechanism instead. Package pinning shows up in the Manifest.toml and is accessible from Pkg.dependencies() values as an is_pinned flag.
From @omus:
The current logic used to determine what packages should be included in the system image uses a file called
TightProject.tomlwhich uses the=compat operator to specify an exact version of package. The logic used to do this is rather frail at the moment and can break. Since Julia already uses Manifest.toml to keep track of an exact list of packages we should make use of that infrastructure to makejulia_podmore reliable and ergonomic.An outline of the proposal:
julia_podusers to specify what packages should be included in the system image. This information is recorded in the project'sManifest.tomlfile.julia_podusers will be unable to accidentally change the package version of something baked into the system image. This provides a nice barrier which avoids confusion between the version reported by Pkg and the version of a package in the system image.julia_podwill read the projectManifest.tomlto find all the pinned packages as well their transitive dependencies. From this we'll generate aSysimgManifest.tomlandSysimageProject.tomlSysimg*.tomlfiles will be COPY'ed. Afterwards we'll instantiate the packages and build a Julia sysimg which includes all of those packages. The next step will COPY theManifest.tomlandProject.tomlinto the Docker container and then instantiate the packages. This should allow for system images to only be re-built when the list of pinned packages changed or the associated versions.JULIA_LOAD_PATH="/project:/sysimg:@v#.#:@stdlib"). Mainly this is just a convenience to be able to view what packages were included in the system image.