Open ldorau opened 2 weeks ago
For fsdax we probably could free pages on destroy (memkind does: https://github.com/memkind/memkind/blob/master/src/memkind_pmem.c#L159). The issue there is performance - deallocating such page is a very expensive operation (since, for fsdax, you are punching a hole in the filesystem) and it may not make sense if we can handle a situation where a provider does not support page deallocation. So by forcing the provider to provide a suboptimal free implementation, we are leaving performance on the table.
For devdax, the situation is a bit different. It's a special case where not all mmap/munmap capabilities exist. For example, you need to align all your requests to the size with which the devdax region was created. This limitation might be tricky to adhere to for pool implementations, since you might want to be able to break down huge pages into smaller pages, and then merge them back again. Memkind does not implement free for devdax "provider".
I guess we could only ever expose devdax/fsdax providers wrapped in a coarse provider. In that case, this would introduce an extraneous abstraction layer for scenarios that use fsdax provider and a jemalloc pool (which has builtin support for providers not support page deallocation).
Finally, we might also eventually create a provider that allocates out user-provided pages. Like fixed malloc in memkind: https://github.com/memkind/memkind/blob/master/examples/fixed_malloc.c#L31 This is basically the same thing as devdax, and we might not want to munmap pages mmaped by the user. Again, we can just wrap this in a coarse provider, but that adds overhead.
Second, in this PR you need to iterate via the tracker (that contains records from all pools) and filter only records for the corresponding pool. It might be a performance impact in real life.
Yes, but I'd argue this should be happening regardless of whether providers are allowed to not support free. Otherwise we risk bloating the tracker data structure if some piece of code does not properly clean up after itself. And it only happens on provider destroy, so should be infrequent and not on the hot path. But I do a agree that having an O(n) operation isn't ideal. We probably should think of how we can use reduce this (with some reverse lookup?).
The File Memory Provider and the DevDax Memory Provider currently do not support the
free()
operation. They should be used together with the Coarse Memory Provider (#715) or with the jemalloc pool manager that can handle such situation.It may seem weird that some memory providers do not support the
free()
operation. This issue is just for the discussion of this topic.Ref: #759
See #759 for the beginning of this discussion: https://github.com/oneapi-src/unified-memory-framework/pull/759#issuecomment-2382742685 :
@vinser52 @bratpiorka @pbalcer