search

spdk / spdk

Storage Performance Development Kit
https://spdk.io/
Other
3.07k stars 1.2k forks source link

SPDK Blobstore support object? #183

Closed hellowaywewe closed 6 years ago

hellowaywewe commented 7 years ago

Hi,

(https://github.com/spdk/spdk/tree/master/lib) from this link, we know SPDK can support device store. And (http://7xweck.com1.z0.glb.clouddn.com/cephdaybeijing201608/04-SPDK%E5%8A%A0%E9%80%9FCeph-XSKY%20Bluestore%E6%A1%88%E4%BE%8B%E5%88%86%E4%BA%AB-%E6%89%AC%E5%AD%90%E5%A4%9C-%E7%8E%8B%E8%B1%AA%E8%BF%88.pdf) from this link, it talked about accelerating Ceph via SPDK.

we know ceph can be in favor of block device store and object store. does SPDK Blobstore support object store or is there some plan to do with object store?

And when version kraken comes, they propose bluestore in ceph. BlueStore can implements lockless, asynchronous and high performance storage service, which is like spdk blobstore. Is there something conflict in ceph bluestore and spdk blobstore?

Could someone offer some help?

benlwalker commented 7 years ago

Someone implemented a block device in Ceph using SPDK, so you can technically use SPDK with Bluestore to accelerate it. I've never used it though, and it wasn't written in conjunction with the SPDK community here.

On the surface, Bluestore and Blobstore are very similar things. They're both highly simplified filesystems that run in user space, and that are primarily used to back RocksDB. Beyond that, I think the similarities end. Blobstore is, as far as I'm aware, much more overtly designed for NVMe devices with ultra low latency and high parallelism, to the point where Blobstore requires atomic write unit of 4KiB and other NVMe-specific features. Bluestore is designed to run on a much wider range of hardware, including hard drives. Further, Bluestore is part of Ceph, whereas Blobstore is part of SPDK and is not integrated into Ceph today.

If someone thought there was value in using Blobstore in Ceph, I'd be supportive of that work, but that effort is not on the current SPDK roadmap.

hellowaywewe commented 7 years ago

1、We know SPDk can be in favor of block store and accelerate ceph block store. Does SPDK support object store? Is there some plan to do with object store?

2、 in spdk/lib/bdev/rbd/bdev_rbd.c, there is a series of workflow for ceph operation, such as:rados_create(cluster,NULL),rados_conf_read_file(cluster,NULL),rados_connect(cluster),spdk_io_channel_get_ctx(ch), etc. I guess spdk did not work in osd backend, rather than in front of osd, am I worng? and when Ceph bluestore comes, How SPDK works on Ceph bluestore?

Could someone share your idea with me?

ghost commented 6 years ago

Regarding point 1, Ceph already has a bluestore backend to use SPDK for storing object data on the OSD: http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#spdk-usage This doesn't require any object API support from SPDK; it is just using the block-oriented APIs that we already provide, and the mapping to objects is handled within the existing Ceph OSD code.

For point 2, the SPDK bdev/rbd driver is using the Ceph RBD (RADOS Block Device) library to access Ceph block devices, which are backed by objects under the hood. See http://docs.ceph.com/docs/master/rbd/ for additional information. The bdev/rbd driver is not related to the OSD backend or exposing objects in the SPDK API; it just uses the existing librbd/librados library to access Ceph block devices.

To answer the overall question, I believe writing a local object (key/value) store on top of SPDK blobstore may make sense; the SPDK blobfs/RocksDB integration is one existing implementation of this idea. Implementing a distributed object storage system is a much larger undertaking, and I think Ceph is probably a good place to start rather than building one from scratch in SPDK.

I'm going to close this issue for now, since I believe the question has been answered, but if you have specific usage questions or bugs, please feel free to open another issue.