[C++] Implement direct support for shared memory arrow columns

I consider the option of memory-mapping columns to shared memory to be valuable. Such option will be triggered if specific metadata are supplied. Given that many data frames backed by arrow are used for machine learning I guess we could somehow benefit from treating differently the data (most likely data buffer columns) that will be fed into the GPUs/FPGAs. To enable such change we would need to address the following issues:

First, we need each column to hold an integer value representing its associated file descriptor. The application developer could retrieve the file-name from the file descriptor (i.e fstat syscall) and inform another application to reference that file or inform an FPGA to DMA that memory-area.

We also need to support variable buffer alignment (restricted to powers-of-2 of course) when initiating an arrow::AllocateBuffer() call. By inspecting the current implementation, the alignment size is fixed at 64 bytes and to change that value a recompilation is required 1.

To justify the above suggestion, major FPGA vendors (i.e Xilinx) benefit heavily from page-aligned buffers since their device memory is 4KB 2. Particularly, Xilinx warns users if they attempt to memcpy a non-page-aligned buffer from CPU memory to FPGA's memory 3.

Wouldn't it be nice if we could issue from_pandas() and then have our columns memory mapped to shared memory for FPGAs to DMA such memory and accelerate the workload? If there is already a workaround to achieve that I would like more info on that.

I am open to discuss any suggestions, improvements or concerns.

Environment: Linux Reporter: Dimitris Lekkas

_{Note: This issue was originally created as ARROW-5069. Please see the migration documentation for further details.}

apache / arrow

[C++] Implement direct support for shared memory arrow columns #21559