FI_HMEM capability check in fi_getinfo

[a-szegel]

Is your feature request related to a problem? Please describe. Allow the provider to tell the user which types of FI_HMEM they support, and the operations they support on each type of HMEM.

Describe the solution you'd like I would like to extend the existing fi_get_info API to allow us to allow providers to say what types of HMEM they support, and what operations they support on each type of HMEM. I want to be able to express that EFA supports Synapse, Neuron, and CUDA (but not ZE or ROCR). I would like to be able to express to the user that we only support HMEM atomics for CUDA, but not the others.

Describe alternatives you've considered The provider must implement all of the libfabric capabilities it claims to support on every HMEM that it initializes in order to resolve ambiguity to the user.

In the future, when we have multiple different types of accelerators attached to an instance, it would be nice to have specific support claims per type of accelerator (vs generic FI_HMEM).

Additional context We may want additional HMEM capabilities like p2p support to give the user a clearer picture of what each provider supports at the fi_getinfo stage.

[shefty]

I think this needs to be a separate query call somewhere, similar to collectives/atomics. The amount of detail here is really too much for a getinfo call.

[lrbison]

To expand a bit more on this: if multiple HMEM types are to be supported in a single fabric, then we probably need to take a close look at the API and how it allows users to understand the capabilities of libfabric on each of those interfaces separately.

Alternatively, should libfabric only support a single HMEM at a time, we should probably take a look at the documentation and ensure that is made clear.

[shefty]

The intent of the SW is to support multiple accelerators simultaneously. I don't know if that's a likely configuration that system admins will adopt in deployments (outside of testing), but I will push that libfabric shouldn't have such a restriction. The internal core implementation around HMEM can support multiple accelerators.

[shefty]

Adding to his, I do expect that a provider should be smart enough to detect what's actually available in the system and adjust accordingly. So, if there's only 1 HMEM available, and the provider does not support it, it should not indicate support for HMEM. Now, if there's 2, and it can only support 1, then we need something more beyond the app getting ENOSYS when it tries to register memory, for example.

[a-szegel]

If there is 1 HMEM available and we can only support a subset of operations on it, but a bigger set of operations on system memory. What should we claim support for?

[a-szegel]

Should that be two info structures... One for HMEM and one for system?

[tschuett]

• Does the query take a domain as a parameter?
• Does the query take hints?

  int fi_query_hmem(struct fid_domain *domain, uint64_t flags, struct fi_hmem_info **info);
  int fi_query_hmem(struct fid_fabric *fabric, uint64_t flags, struct fi_hmem_info **info);