rapidsai / kvikio

KvikIO - High Performance File IO
https://docs.rapids.ai/api/kvikio/stable/
Apache License 2.0
148 stars 53 forks source link

[FEATURE] Connect KvikIO to a File Descriptior #182

Open milesp-nvidia opened 1 year ago

milesp-nvidia commented 1 year ago

I'm requesting the ability to connect a file descriptor, like a socket handle, to Kvikio so the data from that descriptor can be polled from gpu and directly loaded without having to reference back to a CPU data handle. Currently, the CuFile submodule of KvikIO does not support file descriptors.

A similar API to this in python might be the socket module.

Usecase: Data streaming from a socket connection to a waiting process with a fully GPU workload.

madsbk commented 1 year ago

This is definitely something KvikIO could support however it will not use GPUDirect Storage (GDS) since CUDA's cuFile only support files.