This adds support for user buffer registration to the NCCL API.
As this is just a performance optimization/hint, to simplify users' lives, all backends implement this, but may do nothing.
Two points for future improvements:
Right now the NCCL backend does not use registered memory for its internal memory allocations (which show up in custom collective implementations). This is mainly because memory needs to be registered with a communicator, and I am not sure whether it can be shared across communicators.
Our internal interface to the NCCL registration system does not really handle the case where the same buffer is registered with different communicators. This is basically for the same reason.
I intend to fix these once I better understand how registration works with multiple communicators.
This adds support for user buffer registration to the NCCL API.
As this is just a performance optimization/hint, to simplify users' lives, all backends implement this, but may do nothing.
Two points for future improvements:
I intend to fix these once I better understand how registration works with multiple communicators.