UcxShuffleManager - main entry point to the plugin. All logic would be implemented in 3 callbacks:
registerShuffle - called on a driver. Indicates that this job would have a shuffle. Driver allocates metadata buffer, so mappers can publish data and index file addresses and keys.
shuffleBlockResolver - called on mapper, so it'll mmap index and data file and publish it's addresses to driver.
getReader - called on a reducer. All the logic to get shuffle blocks.
UcxShuffleManager - main entry point to the plugin. All logic would be implemented in 3 callbacks:
registerShuffle- called on a driver. Indicates that this job would have a shuffle. Driver allocates metadata buffer, so mappers can publish data and index file addresses and keys.shuffleBlockResolver- called on mapper, so it'll mmap index and data file and publish it's addresses to driver.getReader- called on a reducer. All the logic to get shuffle blocks.