How to specify cxx_binary runtime dependencies?

[Overhatted]

I'm building a cxx_binary that has some runtime dependencies, both in the form of plugin shared libraries (plugin because they are loaded at runtime using dlopen and should not participate in the build) and configuration files. How do I specify this in Buck2?

It seems like cxx_binary's resources is supposed to be used for that but when I use it I only get a resources.json file in the buck-out folder and they are not in the working directory of the binary when I call buck2 run.

Has it simply not been implemented yet? Is the file supposed to be read by an external tool?

Thanks for making Buck2!

[ndmitchell]

The expectation is that the resources.json lives relative to the binary itself. You can't change the working directory of the binary when it runs as that is probably important to the user who ran it, so relative to the binary is the only option. Internally we have a library that looks for the resources.json, loads it etc. That code isn't open source, but it has runtime dependencies on Folly (e.g. folly::parseJson), creates a global singleton etc, so probably wouldn't actually be useful for open source users. It shouldn't be hard to read that file and follow it, but shout if it isn't obvious.

[zjturner]

I'm not sure I follow, so here's a concrete example.

If my build generates an so and an executable, and the executable intends to load the so at runtime and must be in the same working directory as the exe, how do I solve this? A json file isn't going to help me do anything, as I would need to hardcode logic in my executable to explicitly load the so from the location in the json.

This is expecially important on Windows, where it's not easy to load a dll from an arbitrary location on disk, it has to be in the same folder.

What is the solution to this?

[cjhopman]

So most of the resources implementations (not just c++, but also python, java, etc) have both a build time component and a runtime component. For c++, the build time component produces the resources.json file (which iirc is basically a map of name to path) and ensures that all the referenced files are available along with the exe (at the paths in the resources file). The runtime component is a small library that allows finding a file based on the resource name.

I think you could imagine how you could extend that to embed the resources.json and all the resources into a single archive and have the runtime component find the files in the archive.

For your needs, if you want to have just an implicit runtime component (where things just look up files themselves and assume they'll be in specific locations relative to the executable), I think it would mean having a target that depends on the binary (and its resources.json and all resources) and produces the sort of structure that you want for your runtime component.

Now, specifically for c++ binaries and so dependencies, I wouldn't expect these to be represented by resources at all. These should be cxx_library or prebuilt_cxx_library (maybe prebuilt_cxx_library_group?) dependencies and will be handled by that (i think there's what I would consider a bug that if you use the non-default "static" link_style that a c++ binary on windows won't get it's shared lib dependencies laid out with it, but i'm not sure).

[zjturner]

If I understand correctly, you would need a "layout" target that is an cxx_binary or prebuilt_cxx_binary. Then when you run that binary, it reads the resources.json for the dependent binary and copies files around on disk?

That's a little strange, because it means that you can't do buck2 run //target:exe until after you do buck2 run //target:layout-exe. Or am I misunderstanding?

[cjhopman]

No, I mean you'd do the layout at build time. You'd have your cxx_binary be like //target:exe-bin and then some other target //target:exe that depends on it and lays it out on disk as you want. Then you just do buck2 run //target:exe.

[zjturner]

Ok, I think I see what you mean. In this case, is //target:exe a genrule()? Or a custom rule that we have to write? It seems like genrule might not be powerful enough for this out of the box.

[ndmitchell]

You can do most things with a genrule - e.g. make it

genrule(
   name = "exe",
   cmd = "cp $(location //target:exe) $OUT/exe && cp $(location //target:resources) $OUT/resources)",
)

So I think should be enough. You can wrap that in a macro. But a rule might be clearer. If you start with a macro, you can make it into a rule later without changing the users.

[pollend]

just wanted to leave this here as reference

def _package_app(ctx: AnalysisContext) -> list[Provider]:
    files = ctx.attrs.files
    artifacts = []
    for f in files:
        dest = ctx.actions.declare_output(f)
        artifacts.append(dest)
        ctx.actions.copy_file(dest, files[f])

    for res in ctx.attrs.resources:
        info = res[DefaultInfo]
        for f in info.default_outputs:
          dest = ctx.actions.declare_output(f.basename)
          artifacts.append(dest)
          ctx.actions.copy_file(dest, f)

    return [DefaultInfo(default_outputs = artifacts)]

package_app = rule(
    impl = _package_app,
    attrs = {
        "resources": attrs.list(attrs.dep(), default = []),
        "files": attrs.named_set(attrs.source(), sorted = True, default = [])
    },
)

package_app(
    name = "01_Transformation",
    resources = [ 
        "//app:app"
  ],
  files =  {"directory/test/file.txt":"directory/file.txt"}
)