Package init methods called multiple times

[asinghvi17]

Currently, if compiling multiple packages with overlapping dependencies using compile_incremental(packages...), the packages' __init__ methods will be called twice. The current solution is to edit the precompile file; is it possible to fix this in the code generation stage?

Error:

julia> compile_incremental(nothing, "~/Desktop/PackageCompiler/incremental_precompile.jl" |> expanduser |> abspath)
┌ Info: activating new environment at ~/.julia/packages/PackageCompiler/oT98U/packages/Project.toml.
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:519
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
WARNING: using Contour.Contour in module anonymous conflicts with an existing identifier.
ERROR: LoadError: LoadError: Freetype already initalized. init() called two times?
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] ft_init at /Users/anshul/.julia/packages/FreeTypeAbstraction/dlDGC/src/functions.jl:62 [inlined]
 [3] __init__() at /Users/anshul/.julia/packages/FreeTypeAbstraction/dlDGC/src/FreeTypeAbstraction.jl:19
 [4] top-level scope at /Users/anshul/Desktop/PackageCompiler/incremental_precompile.jl:6456 [inlined]
 [5] top-level scope at ./none:0
 [6] include at ./boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1038
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] top-level scope at none:0
in expression starting at /Users/anshul/Desktop/PackageCompiler/incremental_precompile.jl:6455
in expression starting at /Users/anshul/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl:10
ERROR: failed process: Process(`/Applications/Julia-1.1.app/Contents/Resources/julia/bin/julia --optimize=3 --output-o=/Users/anshul/.julia/packages/PackageCompiler/oT98U/sysimg/sys.a -g1 --track-allocation=none --startup-file=no --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --handle-signals=yes --warn-overwrite=no --compile=yes --depwarn=yes --cpu-target=native --track-allocation=none --sysimage-native-code=yes --sysimage=/Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib -g1 --compiled-modules=yes --optimize=2 /Users/anshul/.julia/packages/PackageCompiler/oT98U/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]
Stacktrace:
 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
 [2] pipeline_error at ./process.jl:785 [inlined]
 [3] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
 [4] run at ./process.jl:724 [inlined]
 [5] #run_julia#1 at /Users/anshul/.julia/packages/PackageCompiler/oT98U/src/compiler_flags.jl:225 [inlined]
 [6] #run_julia at ./none:0 [inlined]
 [7] #compile_incremental#62(::Bool, ::Bool, ::Bool, ::Nothing, ::Function, ::Nothing, ::String) at /Users/anshul/.julia/packages/PackageCompiler/oT98U/src/incremental.jl:82
 [8] compile_incremental(::Nothing, ::String) at /Users/anshul/.julia/packages/PackageCompiler/oT98U/src/incremental.jl:79
 [9] top-level scope at none:0

The offending code segments:

# We need to use all used packages in the precompile file for maximum
# usage of the precompile statements.
# Since this can be any recursive dependency of the package we AOT compile,
# we decided to just use them without installing them. An added
# benefit is, that we can call __init__ this way more easily, since
# incremental sysimage compilation won't call __init__ on `using`
# https://github.com/JuliaLang/julia/issues/22910
using Pkg, Observables, Documenter, IntervalSets, MakieGallery, MeshIO, ZipFile, FixedPointNumbers, IterTools, ColorSchemes, Serialization, LinearAlgebra, IndirectArrays, ImageFiltering, StatsMakie, FreeType, AxisArrays, ColorTypes, ColorVectorSpace, AbstractPlotting, ImageMagick, ModernGL, GeometryTypes, Colors, RDatasets, FileIO, ImageCore, StaticArrays, GLFW, FreeTypeAbstraction, ImageAxes, GLMakie, DataFrames, GDAL, PackageCompiler, Printf, ImageTransformations, Markdown
for Mod in [Pkg, Observables, Documenter, IntervalSets, MakieGallery, MeshIO, ZipFile, FixedPointNumbers, IterTools, ColorSchemes, Serialization, LinearAlgebra, IndirectArrays, ImageFiltering, StatsMakie, AxisArrays, ColorTypes, ColorVectorSpace, AbstractPlotting, ImageMagick, ModernGL, GeometryTypes, Colors, RDatasets, FileIO, ImageCore, StaticArrays, GLFW, ImageAxes, GLMakie, DataFrames, GDAL, PackageCompiler, Printf, ImageTransformations, Markdown]
    isdefined(Mod, :__init__) && Mod.__init__()
end

# bring recursive dependencies of used packages and standard libraries into namespace
for Mod in Base.loaded_modules_array()
    if !Core.isdefined(@__MODULE__, nameof(Mod))
        Core.eval(@__MODULE__, Expr(:const, Expr(:(=), nameof(Mod), Mod)))
    end
end

[asinghvi17]

Here's the code in PackageCompiler: https://github.com/JuliaLang/PackageCompiler.jl/blob/fa1f43a56ef7f448a4c9af57dedb37174eaa36f8/src/snooping.jl#L56-L73

[asinghvi17]

Dangit, I keep pressing the wrong button :P

[asinghvi17]

It seems like this is caused by PackageCompiler essentially redoing the entire process for each package?

[asinghvi17]

https://github.com/JuliaLang/PackageCompiler.jl/blob/54c0c1255227c8a94de402b41e05d22ea98b5013/src/snooping.jl#L46-L54 is executed for each package. Perhaps we could set a keyword argument to have the function not do that, letting the calling function handle it?