discuss: Using Apache OpenDAL for more storage services support

inlinedio / ikv-store

High-performance key-value store for ML inference. 100x faster than Redis.

https://inlined.io/

MIT License

201 stars 5 forks source link

discuss: Using Apache OpenDAL for more storage services support #116

Closed Xuanwo closed 4 months ago

Xuanwo commented 4 months ago

👋, I found this project from the github trending. It's seems a cool project.

I noticed that README said ikv support all public storage services but the code only has native s3 support. Do you have interest to integrate with opendal for more storage services support?

OpenDAL is a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services.

I'm the PMC Chair of this project. Please let me know if I can provide help.

pushkarmoi commented 4 months ago

Hi @Xuanwo , at the moment IKV only supports online/streaming writes via its provided client interfaces (in Go and Java). The S3 integration is an IKV internal workflow - for storing and retrieving db images (ex. bootstrapping new hardware). We do not allow users to bulk write into IKV (from varied sources) - as of now. OpenDAL looks very promising and thanks for dropping a note - we will keep it in mind for future.