Object store based metadata store

restatedev / restate

Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.

https://docs.restate.dev

Other

1.65k stars 37 forks source link

Object store based metadata store #1882

Open tillrohrmann opened 3 months ago

tillrohrmann commented 3 months ago

With S3's latest addition to support conditional writes, we can consider building our metadata store based on an object store. The metadata store implementation could store all its state in a single blob on S3. When accessing the data, it could fetch the blob and extract the requested parts.

Xuanwo commented 1 month ago

Hi, I'm from the opendal community and want to promote opendal for implementing an object store-based metadata store. I personally believe it would be a great fit.

OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services.

It's now support nearly all object storage services like S3, Gcs, Azblob, Aliyun OSS, ...

Welcome to check it out; I'm eager to collaborate.

tillrohrmann commented 1 month ago

Thanks a lot for your suggestion @Xuanwo. Opendal looks really interesting and it has come up earlier in some internal discussions as well. Once we get started with this issue, we'll get back to you on it.