Collaborate

[hekaldama]

Hi there.

We are about to open source our Ceph RDB driver plugin and wondered if you all would like to collaborate on this to make one good one?

Let me know and thanks!

[hekaldama]

The code will be found here: https://github.com/yp-engineering/rbd-docker-plugin

[h0tbird]

+1 as a humble watcher

[erikh]

The repository seems blank?

I need to get back to you on the rest.

[erikh]

We're totally open to collaborating, but I think we'll all be happier if we clarify what we mean by it, e.g., what are the goals here. Patches are always welcome, but I'm gathering you mean more than that. What parts of volplugin specifically interest you? Perhaps the conversation is easiest to start there.

volplugin is pretty immature as of yet and we do have our own specific set of goals, but I'm sure we can find something worth sharing. :)

[jainvipin]

+1

[hekaldama]

Code is up and was developed by @porcupie. He liked various things from this project, but wanted some other things as well. We developed ours in house then are releasing it with the hopes to work with each other.

[erikh]

Cool! I'll read this over tonight or tomorrow morning.

[porcupie]

I especially liked that you had a way to test the Ceph portions (using vagrant/virtualbox). I don't have much in the way of test coverage atm and our code is very alpha as well.

I also liked the json config of the various ceph cluster configs for volplugins. As Adam said, ours was built in-house to serve a specific needs, but perhaps we could learn from each other.

Would also be interested in the direction / goals for your project. Still going over your source code as well, so I'm not totally familiar yet with how it works.

[h0tbird]

With the aim to join efforts I want to point out that someone else is working on that too: https://github.com/AcalephStorage/docker-volume-ceph-rbd

[leseb]

First thank you all for your efforts, hopefully every driver writters will see that :). Humble +1 on my side as well.

It'd nice if we could have a single/official volume driver. Ideally it'll live under the Ceph github namespace. I can do the necessary arrangements to create a repo and add core members to it. My point is not to take over anything to anyone but more to strengthen the whole collaboration and to accelerate the adoption of this volume plugin. Having a common place to store the driver is IMO the best we can do. This will avoid unnecessary 'ownership' issues (this will likely be the first thing to come out, where do we store the code etc). Moreover, after looking at the various repositories, it seems that they all aim to the same goal and the way things got implemented is somehow similar. So the idea might be to discuss and/or rediscuss how to merge all the efforts in a new place, select the people who will get ownership of that repo so they can keep on pushing code.

Thoughts? Thanks again :).

[erikh]

Hi folks,

This feels really good to be able to do something so many people are interested in collaborating on.

I need to further clarify our goals with processing storage intent vs. just doing basic ceph storage as volumes. I think it's confusing this a bit; I don't think several of these storage drivers have this feature set in mind.

Fundamentally I see this kind of an issue not as "should we share the code", but more accurately, "what parts of code does sharing make sense for" and even in open source projects as large and popular as docker, that can only be said about a subset of it.

I'd like to identify these parts of code: most of librbd has overlapping functionality with go-ceph, but it seems like the mapping code in librbd might be useful for those of you who do not want to shell out to the rbd tool at all. We could also petition for it to be added to go-ceph, if that author is interested.

The current volplugin tests are a hacked up amalgamation of things ceph needs + things docker needs to make volume drivers work. I don't think they're very reusable. One thing that could be immensely useful is a unified suite for testing volume drivers themselves, but I don't know when I'll have time to get to making something like that, but if someone's interested in doing the leg-work we can definitely make use of it, and I would help you get started.

I'll add more as I think of it. If anyone wants to go through our code, or detail what they think is worth sharing from theirs, echo it here, I'm sure we'll find a set of good projects this way to share code between us.

[h0tbird]

the mapping code in librbd might be useful for those of you who do not want to shell out to the rbd tool at all

I think any CoreOS user will benefit from that because the rbd binary is not present in such systems.

[erikh]

ok, I'll try to take care of that code later today.

[erikh]

Sorry for the delay. I have extracted the librbd code: https://github.com/contiv/librbd

[hekaldama]

@erikh Nice! Thanks for this.

[h0tbird]

Great!

[jainvipin]

+1

[hekaldama]

@noahdesu @aisrael what are your thoughts wrt:

https://github.com/contiv/volplugin/issues/10#issuecomment-132196987

and

https://github.com/contiv/volplugin/issues/10#issuecomment-132262864

I think all of us contributing to a handful of defined shared libs is very profitable. I don't mind if they all end up under ceph/ either.

Looking forward to your thoughts.

[dotnwat]

Hi @erikh I'm the author of go-ceph. The general direction of go-ceph was originally to provide minimalistic go wrappers around the various ceph APIs that could be used to build up higher level APIs or tools. At this point I'm largely just doing reviews, keeping things up to date, and I'm generally open to merging anything that people find useful.

@leseb moving go-ceph under the ceph namespace is fine with me.

[leseb]

Hi @noahdesu I just created https://github.com/ceph/go-ceph, you should be able to push your code there. You will get 'merge' permissions soon.

[erikh]

hi @noahdesu! I plan to implement krbd mapping in your library, would that be an acceptable patch?

[dotnwat]

@erikh this API here https://github.com/ceph/ceph/blob/master/src/include/krbd.h ? Sounds good to me!

[dotnwat]

@leseb thanks. I should already be in the ceph organization so I think that may make it easier.

[leseb]

@noahdesu great! Ping me if you have any issue :)

[erikh]

I am looking for storage experts to query about this plugin. It'd be a very useful way to assist with minimal effort and do me a huge favor.

I wanted to take a minute to clarify what we mean by "processing storage intent". The master/slave approach we are using is very powerful, and we can leverage it as a group if we wished to rally behind it. I know this is a bold proposition. :)

So the architecture is master/slave with the volmaster being the master, and the volplugin being the slave. The master is responsible for communicating intent while the slave is responsible for "realizing" the intent. This is a fancy way of saying it actually executes something based on what it was told.

The approach is intended to enable cluster-wide scheduling of storage. With this we can enable snapshots, garbage collection, all sorts of centralized logic, while maintaining the power of a distributed filesystem.

For example, this is what the volmaster can do now. For a given tenant, it can communicate storage parameters and configure snapshots for each image.

{
  "tenant1": {
    "pool": "rbd",
    "size": 100000000,
    "snapshots": true,
    "snapshot": {
      "frequency": "30m",
      "keep": 20
    }
  }
}

The supervisor for snapshots can be re-purposed for other things, such as a coordinated background fsck or routine garbage collection (cleanup of orphaned images).

The master is very immature (as is the rest of the code) but I think we're on to something. I am hoping that at least a few of you might be interested as well and are looking to contribute to a project. :)

[erikh]

@noahdesu http://ceph.com/docs/argonaut/rbd/rbd-ko/ is the interface I am using -- writing to /sys/bus seems safer. The code is here, but it'll have to be adapted to your system a bit.

https://github.com/contiv/librbd/blob/master/image.go#L107

[dotnwat]

@leseb hmm, I don't seem to be able to push go ceph/go-ceph.

[dotnwat]

It looks like the existing krbd API is also writing to sysfs (https://github.com/ceph/ceph/blob/master/src/krbd.cc), but would have the benefit that it is always implements the up-to-date protocol. I admit I don't follow krbd development, so I may be completely wrong. Either way, I'm open to krbd merges :)

[erikh]

I explored that but the C++ is what holds me back, at least in Go... unless you know of a way. I would prefer to take that route as well.

[erikh]

While I'm thinking of it: the C interface also does not handle the toml/ini style nested configuration, only the top level stuff. If you know of a way to access it, can you let me know?

[dotnwat]

It looks like krbd already has a C interface (https://github.com/ceph/ceph/blob/master/src/include/krbd.h), so that should work directly in Go without any modifications. Are there pieces of the C++ implementation that would need a corresponding C interface for Go? I'm trying to understand how all the pieces fit together. Can you point me at an example of where the toml/ini style configurations are used to interact with krbd (perhaps the toml/ini config handling could remain in Go)?

[erikh]

Sorry, I'm communicating poorly.

I'll look into the krbd C interface. I guess I must have missed that header.

As for the config files, stuff like this in the C interface is not handled by the source, best I can tell:

[client.admin]
        key = <some key>

I cannot find a way, with the C interface, to get at this key.

[dotnwat]

@erikh I took a quick look at the krbd implementation. It appears to me that when you create a krbd context (krbd_create_from_context(rados_config_t cct, struct krbd_ctx *pctx)) that things like the key from the configuration file are automatically extracted (via the cct parameter), and that rados_config_t can be programmatically constructed, populated from environment variables, or a ceph.conf. So I think we should be able to use go-rados (func (c Conn) ReadConfigFile(path string) error) to build a context and then pass that context to krbd which would extract configuration values.

Let me know what you think. I'm totally fine with that as a longer term thing, and now just merging your current API implementation into go-ceph.

[erikh]

Nah, I can do that this week. No need to do it twice.

[leseb]

@noahdesu can you at least PR?

[leseb]

@noahdesu you should be good to go now :)

[dotnwat]

@leseb looks good thanks

[porcupie]

@erikh: Though it does not involve using ceph libs and using @noahdesu's suggestion seems like the better way to go (rados context that can import ceph conf file, env vars, etc) -- before, I was looking at parsing the INI style configuration files using this project: https://github.com/vaughan0/go-ini - which can import into a Map and can even support keys with spaces (e.g. "mon addr = 10.1.1.5:6789"). You lookup the value using the "[section]" and "key" name. Though I think ceph supports spaces and underscores for key names, making it easier to support other ini parsers by converting your conf to underscore delimiters.

[erikh]

@porcupie the krbd code should suffice, but I'll keep digging if it does not. Thanks for the update.

[erikh]

@noahdesu the code isn't a part of the librados or librbd distributions so it can't be used :(

I'm going to port librbd's mapping code for now. I'll try to see if I can figure out the config issue that is raised by this.

[dotnwat]

@erikh oh bummer, I see now that libkrbd is a dependency for the rbd tool, but isn't installed, which makes sense. if you think that libkrbd is something that would be good to expose, it's probably a good idea to start that process for the next release cycle.

[dotnwat]

just porting the mapping code for now sounds good.

[erikh]

OK. I'll see what's involved there, I agree that is the best approach.

For now to solve the config problem, I can see this going two ways:

• provide a struct which the user must populate themselves
• integrate https://github.com/burntsushi/toml to handle the config parsing.

Which would you prefer? Also, I'll start an issue on ceph/go-ceph to discuss this further in a few minutes here.

[dotnwat]

Is it possible to do everything through the rbd command line tool? If so, then running a shell command seems like a reasonable thing to do since it would be indirectly constructing all the sysfs commands via libkrbd.

[erikh]

Ok, I'll do that for now. Thanks

[dotnwat]

Just to be clear, I was thinking that you could keep your Go API and then do something like out, err := exec.Command("rbd <args>").Output() as an alternative to constructing the sysfs commands so that later when libkrbd is available, the API could remain unchanged. If using rbd cli also doesn't work, then I think that I'm neutral on the two options you proposed.

[erikh]

I worry it's going to be racy, but I'm trying porting to the CLI first.

[erikh]

Hey folks, a lot has changed in this project in the last month. I'm going to close this, but please feel free to file pull requests or otherwise suggest ideas where we can work together. Thanks for your interest!