Add ceph rados datastore

ipfs / go-datastore

key-value datastore interfaces

MIT License

228 stars 64 forks source link

Add ceph rados datastore #101

Open bjzhang opened 5 years ago

bjzhang commented 5 years ago

Currently ipfs support s3 and swift for clustering storage. We(ipfsbit) implement a ceph backend through rados(a reliable autonomic distributed object store). The basic function works:

It will be easy to make more optimization if implement such dedicated ceph backend. So, How do we contribute it to ipfs? We hope we could send the patches early to get more feedbacks. Thanks in advance.

magik6k commented 5 years ago

Nice!

Sorry for the delay - the general flow of 'mainlining' a datastore would be:

Put it in a repo somewhere on github, preferably under MIT license (or something compatible with MIT). We can open a repo under ipfs/go-ds-rados (assuming this name makes sense) and give you push access. This is preferred because it gives us easy access in case of refactors.
Publish GX packages for it and it's dependencies (we can help with that as it tends to be painful or at least undocumented)
Open PR in go-ipfs like this one: https://github.com/ipfs/go-ipfs/pull/5516

bjzhang commented 5 years ago

Put it in a repo somewhere on github, preferably under MIT license (or something compatible with MIT). We can open a repo under ipfs/go-ds-rados (assuming this name makes sense) and give you push access. This is preferred because it gives us easy access in case of refactors.

Cool, this name is what I used. Once you create this project, I could push the initial version to it.

Publish GX packages for it and it's dependencies (we can help with that as it tends to be painful or at least undocumented)

I touch gx when I work on go-unixfs, I will take a look other project and ask questions when I stuck.

Open PR in go-ipfs like this one: ipfs/go-ipfs#5516

Got it. I will do it after finish the gx relative work.

bjzhang commented 5 years ago

Hi, we already test go-ds-rados with go-ipfs. Could you please create a repo such ipfs/go-ds-rados? As you said, it is easy to discuss about go-ds-rados specific issues. Hope we could do more to improve the datastore as well as go-ipfs/libp2p. Thanks.

benaryorg commented 1 month ago

If you're like me and you know you've seen the go-ds-rados repo before but can't find it, here's the link.

(sorry for the issue necro-ing, I just spent 10 minutes finding that repo)