Relocatability issues with services files

[JackDunnNZ]

When using AWS.jl in conjunction with PackageCompiler, there are some relocatability issues that come up if you try to load a service that was not included in the generated system image. This is because @service tries to load a path relative to @__DIR__ which may no longer exist.

This is easy to fix by using RelocatableFolders.jl which was designed to solve this sort of problem (see https://github.com/JackDunnNZ/AWS.jl/commit/c5ed3e0f01f83e0bed85bebf4a39e37dc0ae81ed), however my understanding is that this is going to cause all of the services files to be loaded and saved in memory when the package is loaded in case they are needed later, which would be a heavy-handed fix to the problem, and if so a better solution would be needed

[quinnj]

In the script you give to PackageCompiler, can't you just include:

using AWS: @service
@service S3

for whichever service you need? That should load just the service you need and avoid the need to lookup anything at runtime.

[JackDunnNZ]

Yes, that works. The issue comes when a user of the system image tries to load a different service themselves. The context being that a prebuilt system image is distributed to users, where the system image has code that depends on one AWS service but users might want to also use other services themselves. So we don't know upfront which services might be loaded after compilation

[mattBrzezinski]

Sorry I've been offline for the last 3 weeks and am just coming back around to things. What we could do here is re-write the macro to something like,

macro service(module_name::Symbol, p=@__DIR__, features...)

However this would be a breaking change, which I don't think we're quite ready yet to make. I also can't seem to get this to work with the existing macro for backwards compatibility.

[fatteneder]

As already pointed out with the PackageCompiler example, this behavior is against the Pkg.jl's best practices: https://pkgdocs.julialang.org/v1/creating-packages/#Best-Practices

Packages should, in general, not assume that they are located in a writable location (e.g. if installed as part of a system-wide depot) or even a stable one (e.g. if they are bundled into a system image by PackageCompiler.jl).

Starting with v1.11 this will also affect pkgimages, because we are trying to make them relocatable. The ultimate goal is to have Pkg.jl ship those pkgimages over the wire, but this won't work with absolute paths in the source code.

A related issue already popped up in https://github.com/JuliaLang/julia/issues/53859#issuecomment-2027352004, but this is a regression in base Julia and has to be fixed upstream.