This issue is intended to review the exposed interfaces for the resource API, with the intention to understand what needs to be exposed (and what does not). I'll tackle this from the standpoint of our primary use case (at least for now) - the fluence plugin that uses the Go bindings and other TBA out of tree plugins that use fluxion.
You can find the Go bindings here for each of reapi_cli and reapi_module
Level 1: Go Modules
The entrypoint for a Go plugin using the Fluxion Go bindings would be these modules. I don't know the subtle distinction between the cli and module.
reapi_cli imports reapi_cli.h, and uses these functions / structures from it:
struct_reapi_cli_ctx_t: for the context
reapi_cli_new create the new client (context)
reapi_cli_destroy for Destroy
reapi_cli_initialize to init Fluxion with resource graph
reapi_cli_match_allocate for MatchAllocate
reapi_cli_update_allocate for UpdateAllocate
reapi_cli_cancel for Cancel
reapi_cli_info for Info
reapi_cli_stat for Stat
reapi_cli_get_err_msg for GetErrMsg
reapi_cli_clear_err_msg for ClearErrMsg
Level 2: C/C++ bindings
The Go modules above use the following header files and associated c code. It looks like the C code actually imports some C++ headers / code as well! I didn't know you could do that, TIL! I am guessing this is because we are using cgo? And cgo needs to use C, and fluxion is in C++, so we created the c bindings as an intermediate interface?
reapi_cli.h
The TLDR here is that the reapi_cli.h imports some of the C++ bindings, and it's these C++ bindings that bring in different components from resource (higher up) along with jobinfo and this is why we need to include those shared libraries.
I'll stop there because it starts to look like everything is using everything else in resource!
reapi_module.h
TLDR: my impression here is that the distinction might be that this is indeed intended to be a module, meaning it doesn't bring in all the libraries from libresource.so. Was it the case that this module was started but it couldn't meet all the needs that we wanted, so it was left (and then the fluxcli started?)
To step back - let's have a discussion about what our goals are (e.g., to create more separation between the API via more scoped functions? To simplify logic / types to be shared between libresource.so and libreapi_cli.so so they can use the same thing (but we expose a much smaller interface?) Let me know if you want me to dig deeper into any of the above - I'm hoping the links to the top level files help you explore as I did.
Example Use Case
Right now to build an out of tree plugin using fluxion (with the Go bindings) I need to both compile and then have the paths to the (non system installed) .so files exposed via LD_LIBRARY_PATH. Here are those things, and I'll show as a diff for what I have to do currently (red) vs what I'd like to do (green).
And maybe fluxion-resource and reapi_cli are differently named / combined, I'm not sure. But the distinction is that I don't need to tell the linker about paths in the source code of a built flux-sched. They are in default paths somewhere on my system (likely in a container).
Runtime
# These are the flags
# This is what I need to export before I run my binary
+ This should be entirely unnecessary if flux (with the reapi_cli) is installed to a system location like `/lusr/lib`
- export LD_LIBRARY_PATH=/usr/lib:/opt/flux-sched/resource:/opt/flux-sched/resource/reapi/bindings:/opt/flux-sched/resource/libjobspec
# Running the binary!
./bin/icecream -spec icecream.yaml
Questions
And some questions that I have:
I think we should document somewhere when I'd want to use reapi_cli vs reapi_module
Given the existence of the two, is it the case that we really want one thing (that looks more like the module in terms of what it brings it) but that was impossible to implement?
Consistent function naming - e.g., full names vs not. It looks like we are using full names "MatchAllocate" so I think "GetErrMsg" should be "GetErrorMessage" (as an example).
Almost forgot! Ping @trws and @milroy from discussion today.
This issue is intended to review the exposed interfaces for the resource API, with the intention to understand what needs to be exposed (and what does not). I'll tackle this from the standpoint of our primary use case (at least for now) - the fluence plugin that uses the Go bindings and other TBA out of tree plugins that use fluxion.
Level 1: Go Modules
The entrypoint for a Go plugin using the Fluxion Go bindings would be these modules. I don't know the subtle distinction between the cli and module.
reapi_module
reapi_moduleimportsreapi_module.h, and uses these functions / structures from it:struct_reapi_module_ctx_: the contextreapi_module_newused in Initreapi_module_destroyused in Destroyreapi_module_match_allocateused in MatchAllocatereapi_module_infofor Inforeapi_module_cancelfor Cancelreapi_cli
reapi_cliimportsreapi_cli.h, and uses these functions / structures from it:struct_reapi_cli_ctx_t: for the contextreapi_cli_newcreate the new client (context)reapi_cli_destroyfor Destroyreapi_cli_initializeto init Fluxion with resource graphreapi_cli_match_allocatefor MatchAllocatereapi_cli_update_allocatefor UpdateAllocatereapi_cli_cancelfor Cancelreapi_cli_infofor Inforeapi_cli_statfor Statreapi_cli_get_err_msgfor GetErrMsgreapi_cli_clear_err_msgfor ClearErrMsgLevel 2: C/C++ bindings
The Go modules above use the following header files and associated c code. It looks like the C code actually imports some C++ headers / code as well! I didn't know you could do that, TIL! I am guessing this is because we are using cgo? And cgo needs to use C, and fluxion is in C++, so we created the c bindings as an intermediate interface?
reapi_cli.h
The TLDR here is that the reapi_cli.h imports some of the C++ bindings, and it's these C++ bindings that bring in different components from resource (higher up) along with jobinfo and this is why we need to include those shared libraries.
Imports:
flux/core.hresource/reapi/bindings/c++/reapi_cli.hppmatch_op_t, as an example?resource/reapi/bindings/c++/reapi_cli_impl.hppresource/schemaandresource/configand at this point it's going everywhere in resource, e.g., plannerI'll stop there because it starts to look like everything is using everything else in resource!
reapi_module.h
TLDR: my impression here is that the distinction might be that this is indeed intended to be a module, meaning it doesn't bring in all the libraries from libresource.so. Was it the case that this module was started but it couldn't meet all the needs that we wanted, so it was left (and then the fluxcli started?)
Imports:
flux/core.hresource/reapi/bindings/c++/reapi_module.hppresource/reapi/bindings/c++/reapi.hppresource/reapi/bindings/c++/reapi_module_impl.hppresource/reapi/bindings/c++/reapi_module.hppNext Steps
To step back - let's have a discussion about what our goals are (e.g., to create more separation between the API via more scoped functions? To simplify logic / types to be shared between libresource.so and libreapi_cli.so so they can use the same thing (but we expose a much smaller interface?) Let me know if you want me to dig deeper into any of the above - I'm hoping the links to the top level files help you explore as I did.
Example Use Case
Right now to build an out of tree plugin using fluxion (with the Go bindings) I need to both compile and then have the paths to the (non system installed) .so files exposed via
LD_LIBRARY_PATH. Here are those things, and I'll show as a diff for what I have to do currently (red) vs what I'd like to do (green).Compiling
And maybe fluxion-resource and reapi_cli are differently named / combined, I'm not sure. But the distinction is that I don't need to tell the linker about paths in the source code of a built flux-sched. They are in default paths somewhere on my system (likely in a container).
Runtime
Questions
And some questions that I have:
Almost forgot! Ping @trws and @milroy from discussion today.