Expose memspace and policy API for managing host memory

[igchor]

A memspace is an ordered collection of one or more memory targets. It can represent a certain class of memory sources like all NUMA nodes in the system or contain memory targets with different properties (e.g., latency).

A memory target represents a single memory source (e.g., NUMA node) and can have certain attributes (e.g., capacity).

Memspaces to expose (should be exposed as functions or a single function accepting an enum to avoid problems with exporting variables on windows):

• [x] UMF_MEMSPACE_HOST_ALL
• [x] UMF_MEMSPACE_HBW
• [x] UMF_MEMSPACE_LOWEST_LATENCY
• [x] UMF_MEMSPACE_HIGHEST_CAPACITY
• [x] UMF_MEMSPACE_HIGHEST_BANDWITH

Memspace attributes to expose:

• [x] Capacity
• [ ] Latency (as distance to local node / other nodes)
• [ ] Bandwidth
• [x] Numa node id

Memspace management API:

• [x] Filtering based on one (or more) of the attributes
• [x] Sorting based on one (or more) of the attributes
• [x] Iteration over memory targets within a memspace

All memspaces should be a part of libumf (not separate libraries).

Memspace, along with a policy, can be used to create pools and providers (umfPoolCreateFromMemspace and umfMemoryProviderCreateFromMemspace). Implementation of umfPoolCreateFromMemspace should select the best-suited pool based on memspace and policy type and availability of memory pools.

Policies to expose:

• [ ] UMF_MEMSPACE_POLICY_STRICT
• [ ] UMF_MEMSPACE_POLICY_PREFERRED
• [ ] UMF_MEMSPACE_POLICY_BLOCKED
• [ ] UMF_MEMSPACE_POLICY_INTERLEAVED
• [ ] UMF_MEMSPACE_POLICY_LOCAL

Optional:

• [ ] expose system-level memory metrics for memspaces (.e.g total available/free/allocated memory)

[bratpiorka]

@igchor @vinser52 How we should name the default memspace (one that contains all available NUMA nodes)? "host" or "default"? Should we add an additional, dram-only memspace "REGULAR" (named as in the Memkind - "Regular means general purpose memory from the NUMA nodes containing CPUs")? What about HUGETLB and KMEM_DAX?

[igchor]

@igchor @vinser52 How we should name the default memspace (one that contains all available NUMA nodes)? "host" or "default"? Should we add an additional, dram-only memspace "REGULAR" (named as in the Memkind - "Regular means general purpose memory from the NUMA nodes containing CPUs")? What about HUGETLB and KMEM_DAX?

I think the 'default' is fine - as was agreed on the PR.

• HUGETLB: I think this should be just a policy, not a separate memspace.
• KMEM_DAX: I'm not sure - user can easily build it by filtering the DEFAULT memspace using a property (like is_persistent or is_cxl_attached) and that way is more generic.

[vinser52]

Do we want to implement the same set of predefined memspaces like we have in Memkind (as kinds)? My current understanding is that kind in Memkind incapsulates memory storage and policy. So the set of memspaces in UMF will be smaller than a set of kinds. We can start from reviewing what we have in memkind and decide on naming schema for memspaces.

How we should name the default memspace (one that contains all available NUMA nodes)? "host" or "default"?

To be honest both names are not obvious or descriptive. Maybe "host_all"? choosing between "host" or "default" I prefer "default".

Should we add an additional, dram-only memspace "REGULAR" (named as in the Memkind - "Regular means general purpose memory from the NUMA nodes containing CPUs")?

I think yes, but again naming is not descriptive. Need to think about better name.

What about HUGETLB and KMEM_DAX?

Agree with @igchor about HUGETLB. Regarding KMEM_DAX, as a first step we agreed to have only predefined memspaces (so we cannot filter from DEFAULT). In general, I think we should focus on OpenMP use cases and expose memspaces needed for them.