search

JuliaLang / PackageCompiler.jl

Compile your Julia Package
https://julialang.github.io/PackageCompiler.jl/dev/
MIT License
1.4k stars 186 forks source link

include julia.h with shared libraries #884

Open rekabrnalla opened 9 months ago

rekabrnalla commented 9 months ago

A bug was found in julia where @ccallable functions allocate memory for each argument triggering garbage collection. For a high-throughput loop this is a no go. It was observed that @cfunction function pointers do not allocate per each argument.

https://github.com/JuliaLang/julia/issues/51894

Unfortunately for the shared-library use case with the PackageCompiler.jl using @cfunction interface requires a julia install because the "julia.h" include file is not available to shared libraries created with the PackageCompiler.jl.

Can julia.h be made available to shared libraries made with PackageCompiler.jl so that additional embedded functionality can also be accessed via embedded C julia commands? This would save an installation of Julia being required to link in with the shared library.

sloede commented 9 months ago

You can, by populating the julia_init_h_file option, https://github.com/JuliaLang/PackageCompiler.jl/blob/466aec351033ff6ec319e3703a6f8e4ac77fe0ac/src/PackageCompiler.jl#L959-L961 with appropriate values. For example, to include the default init file header plus the Julia header file, you could try something like


julia_init_h_file = [PackageCompiler.default_julia_init_header(),
                     joinpath(Sys.BINDIR, Base.INCLUDEDIR, "julia", "julia.h")]
rekabrnalla commented 8 months ago

Thanks. I will try that. I couldn't find it in the documentation anywhere.
Will it automatically pull in all of the julia.h include dependencies?

sloede commented 8 months ago

I couldn't find it in the documentation anywhere.

It is considered an "advanced" option and thus only documented in the References section. I'd be open to adding a section on custom header/init files in the main docs though.

Will it automatically pull in all of the julia.h include dependencies?

No, you're right. You'd have to do this manually. I also see currently no sane way to automate this, since you'd have to transitively also include those headers. Except maybe to include the entire joinpath(Sys.BINDIR, Base.INCLUDEDIR, "julia") folder in the library bundle.

rekabrnalla commented 8 months ago

So it seems like this is still a valid issue? To somehow make this more manageable, or specify which functions to rehost in the shared library somehow.

sloede commented 8 months ago

Yes, I still consider this to be a valid issue. However, it is unclear to me if such a functionality should be included in PackageCompiler.jl to be automated, or whether it is something that could be documented but would be left up to the user to do manually.

Maybe @KristofferC has on opinion here?