Open joa-quim opened 7 months ago
Looks like, partly, new invalidations in GMT.__init__()
? (and therefore possibly in other things too)
Maybe similar to #53511 ?
Would be interesting to put back all stdlibs into the sysimage and re-time this to see if the effect is purely for moving out stdlibs or if there are other reasons as well.
Would be interesting to put back all stdlibs into the sysimage and re-time this to see if the effect is purely for moving out stdlibs or if there are other reasons as well.
We should consider building this and releasing it as an artifact. It would be using at least for testing and some loading sensitive applications as well.
Worst offender in the precompiled size I have seen is https://github.com/Gnimuc/GLTF.jl. v1.11.0-beta1 (211 MB) is about 5x bigger than v1.10.3 (43 MB).
As you showed in https://github.com/quinnj/JSON3.jl/issues/279, the issue there seems to be egregious use of @inline
on large functions. It isn't obvious why that would change between 1.10 to 1.11 but one reason could be that we are better at precompiling now so more of the (very big due to `@inline) functions get saved to the image file.
This seems to be more or less fixed on 1.11 backport branch. This is using the master branch of GMT.jl:
julia> @time using GMT
0.638436 seconds (668.47 k allocations: 51.051 MiB, 5.54% gc time, 2.08% compilation time)
julia> VERSION
v"1.10.3"
julia> @time using GMT
0.657463 seconds (730.44 k allocations: 49.958 MiB, 1.27% gc time, 1.48% compilation time)
julia> VERSION
v"1.11.0-beta1.40"
But the precompiled image is still ~50% larger.
I did some time ago ans spent quite some effort trying to improve the situation (forgot many of the details). But now I can't even use that package anymore.
| | |_| | | | (_| | | Version 1.10.0 (2023-12-25)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> cd("c:/v"); @time using GMT
Precompiling GMT
1 dependency successfully precompiled in 44 seconds. 87 already precompiled.
46.397674 seconds (4.63 M allocations: 340.133 MiB, 0.28% gc time, 1.79% compilation time)
julia> using PkgCacheInspector
julia> info_cachefile("GMT")
ERROR: Error reading package image file.
| | |_| | | | (_| | | Version 1.11.0-beta1 (2024-04-10)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PkgCacheInspector
julia> info_cachefile("GMT")
ERROR: MethodError: no method matching parse_cache_header(::IOStream)
The function `parse_cache_header` exists, but no method is defined for this combination of argument types.
But note that this cache size issue is common to other packages. For example Makie cache is 60% larger in 1.11 vs 1.10
That last error is due to #49866 in which the signature of an internal method was changed, which is used by PkgCacheInspector.jl. Should be straightforward to fix, will make a PR later.
1.11:
Contents of /Users/kristoffercarlsson/.julia/compiled/v1.11/GMT/EoU0j_u25Qm.dylib:
modules: Any[GMT.Gdal, GMT.Drawing, GMT]
init order: Any[GMT.Gdal, GMT]
1503 external methods
22730 new specializations of external methods (Base 80.0%, Base.Broadcast 14.5%, Base.Iterators 2.5%, ...)
1371 external methods with new roots
36342 external targets
28454 edges
file size: 92678656 (88.385 MiB)
Segment sizes (bytes):
system: 25084036 ( 29.27%)
isbits: 57449436 ( 67.04%)
symbols: 149474 ( 0.17%)
tags: 410413 ( 0.48%)
relocations: 2523569 ( 2.94%)
gvars: 44312 ( 0.05%)
fptrs: 33536 ( 0.04%)
1.10:
Contents of /Users/kristoffercarlsson/.julia/compiled/v1.10/GMT/EoU0j_Vg0I0.dylib:
modules: Any[GMT.Gdal, GMT.Drawing, GMT]
init order: Any[GMT.Gdal, GMT]
1503 external methods
40014 new specializations of external methods (Base 72.8%, Base.Broadcast 14.0%, GMT 5.9%, ...)
1157 external methods with new roots
31428 external targets
24264 edges
file size: 56215744 (53.612 MiB)
Segment sizes (bytes):
system: 18509732 ( 36.03%)
isbits: 30370140 ( 59.12%)
symbols: 150178 ( 0.29%)
tags: 271567 ( 0.53%)
relocations: 1986075 ( 3.87%)
gvars: 50072 ( 0.10%)
fptrs: 30080 ( 0.06%)
Haven't done any more analysis than that. Seems a bit strange that 1.11 has way fewer specializations but still larger file size.
I did some time ago ans spent quite some effort trying to improve the situation (forgot many of the details). But now I can't even use that package anymore.
@joa-quim Please update PkgCacheInspector.jl
to v1.0.1
and try again. https://github.com/JuliaRegistries/General/pull/106291
Thanks. It works now ... but only once. Anyway, I don't know how to use the info to understand what makes the cache file so big.
julia> using PkgCacheInspector
julia> x = info_cachefile("GMT")
Contents of C:\Users\j\.julia\compiled\v1.11\GMT\EoU0j_tYaDV.dll:
modules: Any[GMT.Gdal, GMT.Drawing, GMT]
init order: Any[GMT.Gdal, GMT]
1503 external methods
22890 new specializations of external methods (Base 80.4%, Base.Broadcast 14.0%, Base.Iterators 2.5%, ...)
1384 external methods with new roots
36769 external targets
28713 edges
file size: 102150656 (97.418 MiB)
Segment sizes (bytes):
system: 25201556 ( 29.22%)
isbits: 57871532 ( 67.10%)
symbols: 147304 ( 0.17%)
tags: 412695 ( 0.48%)
relocations: 2534014 ( 2.94%)
gvars: 44560 ( 0.05%)
fptrs: 33416 ( 0.04%)
julia> x = info_cachefile("GMT")
ERROR: Error reading package image file.
THe compile time is quite variable but the differences are comparable to this.
v1.11 cache file -> ~89.5 MB v1.10 -> ~59.5 Mb
Load times