ZFS Support

[FireDrunk]

Is there any support for ZFS planned? Since ZFS is getting more and more integrated in Linux, it would be nice to get some basic information / statistics from storaged.

I'm asking this here, because if storaged implements it, we can integrate it into Cockpit! (www.cockpit-project.org)

[phatina]

Sure! Thanks for the tip. I will add this to my TODO list and talk to Cockpit guys.

[FireDrunk]

Sweet! Shout if you need any testing done!

[FireDrunk]

Any update? I'd be happy to help.

[phatina]

We need to resolve the licensing problem with ZFS...

[FireDrunk]

Why? We won't ship ZFS? Only thing to do is code against libzfs?

[phatina]

We would need to put the module for Storaged providing D-Bus API for ZFS in a separate repo. We can't link to libzfs (CDDL license, right?).

[phatina]

If we link with CDDL code, we break GPL license which is used for this project. Anyway, having support for ZFS filesystem isn't a priority for us. But, if you are willing to join the development, patches are welcomed and we will consider including the code (somehow).

[dreamcat4]

@phatina According to Richard Stallman, who created this whole unnecessary legal mess in the first place. "Linking is not allowed" under the terms of the GPL, because linking makes it a combined work. This he clarified as recently as April 2016.

https://www.fsf.org/licensing/zfs-and-linux

"if you distribute modules meant to be linked together by the user, you have made them into a combined work, and you must release the entire combined work under the GNU GPL."

However, simply reading down to the next paragraph, the section directly beneath says:

"Copyright Holders Can Explicitly Authorize Additional Uses"

So... once the copyright holder (of this project), or a legally authorized representative of them (for this project), who can (and probably should, given the general nature of this project, where using APIs to other things is pretty difficult to get away from), simply modify the existing licence (again, of this project), with the following exception clause:

In this specific case, where the lib to link to is under CDDL (zfs), the most appropriate thing to do is:

https://en.wikipedia.org/wiki/GPL_linking_exception

So if you can please add the preferred linking exception as per above ^^. Then we can make a contribution which links directly... and avoids any mess.

If you can't do that for whatever reason(s)... which I really do not care to know about! Then I believe you are saying we have to make another API? To talk to the zfs api over DBus instead. To avoid you guys of having to add a linking clause to your license? Is that correct?

If this project explecity wants us to introduce the extra overhead / latency of the DBus messaging interface, for no technical reason, or other reasons, except to avoid a self-imposed software license?

Well then we need a clear answer (option a or option b), before making that contribution! Perhaps you can investigate further (discuss with other project members), and get back to us? Kind Regards.

[vpodzime]

Hello guys, first of all -- thanks for not giving up on this! Here's my take on this:

1. Adding a linking exception to udisks would be a change of license. And we cannot really do that on a project like udisks in this phase. There are many copyright holders, authors, etc. and it would definitely cause problems in some distributions with strict licensing policies. So that's clearly a no-go.

2. A DBus API for ZFS would be beneficial to other projects too, not just for udisks. If nothing else, it's a language-agnostic API. The overhead and latency of DBus are not bad, actually, especially for storage management technologies, that doesn't really mean multiple changes every second. In fact, with libblockdev and libblockdev-lvm-dbus, udisks is now able to work with LVM by means of its new DBus API. So not really a big deal. On the other hand, I can see the issue with writing big amounts of code for this all.

3. There are other ways to avoid the licensing issues. One possible way is to use dlopen() and load the library in runtime instead of linking. Which is of course quite cumbersome. The other option is to use the CLI utilities. Which is what udisks does (and has to do due to lack of APIs) with many other things like MD RAID, Btrfs,... anyway (even though in some cases by means of libblockdev).

Would using the CLI utilities be a big issue? I think it's the least cumbersome and least problematic option given the circumstances.

Thanks!

[dreamcat4]

4. Calling out the zfs CLI utilities instead

Yeah! Sounds like a pretty good idea to me.

Then... it would be nice not to lock out the possibility of a future 2nd contribution offer to replace the zfs cmdline calls with a better implementation in DBUS or zfslib C lib later on.

I.e. shelling out to cmdline in thoughtful enough way to not depend upon complex quirks in a ZFS utility. If those features are missing (or difficult to access) via the zfslib C library.

So its good idea to study the zfslib anyhow. And compare the zfslib to what we want to call on the commandline. Make sure it exists in both APIs.

[FireDrunk]

There are talks about JSON output for ZFS, which makes parsing everything on the commandline much easier.

https://github.com/zfsonlinux/zfs/issues/740

Unfortunatly, it's not ready. Parsing the current cmdline output, might be cumbersome, because of tabs, spaces, headings etc.

It mostly depends on what we want udisks to support. If it's only detection and information, it might be feasible, but if we want control, it's probably a looot more work.

[dreamcat4]

Parsing the current cmdline output, might be cumbersome, because of tabs, spaces, headings etc.

Yeah I was just thinking the same thing too. OK so in regards to zfsonlinux/zfs#740 they moved that over to PR zfsonlinux/zfs#3938 .... the current status for JSON support over there is not-quite finished --> so its not been merged:

https://github.com/zfsonlinux/zfs/pull/3938#issuecomment-312110729

[dreamcat4]

Just a quick note about that: I'm not against using JSON at all. From an api standpoint its better than cmdline. However even once the PR for it in zfs gets merged we would then have to also incurr another wait, for the merged feature to then get into the next release before it becomes generally available. And that by itself seems to take a fair while (the zfs releases).

[FireDrunk]

I personally think some form of linking would be the better option (if we disregard the licensing issue for the sake of argument), because the command line might change, but the libzfs implementations will probably be a lot more stable. Since we link directly against specific libraries, functions like detecting whether a disk is part of a ZFS pool, (which might be one of the most important functions) are easy to execute.

I can't think of any other simple option when you take the licensing issue into account :(

[pothos]

Is there a legal difference between starting a process to communicate via JSON vs. dlopen a library to commuicate via C calling conventions? Instead of dlopen it could also be a stub supplied to have the normal linking at compile time. Isn't the more interesting question whether you ship it together as a single binary which would not be the case with dynamic linking against a (low priority) stub?

[dreamcat4]

Isn't the more interesting question whether you ship it together as a single binary which would not be the case with dynamic linking against a (low priority) stub?

• This project would still need to add the linking clause to its existing GPL license.

The linking clause was not invented by Stallman. But by another OSS project. Who had simply had enough of all this nonsense.

It also depends if we can call dlopen optionally at time of program initialization. To not need to compile-in or compile-out zfslib. Then its technically doable.

Its been assumed here the remaining sticking issue is for some of the distros because of their pkg licensing policies. That is according to earlier comment here. Of course I'm not aware which distros have the problem so please provide that information if you know it.

[benyanke]

Perhaps I'm missing something, but the web involves all sorts of multi-license tools connectng. Why not just make a small stub in it's own repo, which can be called by this one? Heck, make it even call over HTTP if we really need to, such as by REST.

Or, to put it another way, how would a simple implementation of this differ from simple bash pipes? There's no issue with running a shell script which pipes a non-GPL program into a GPL program...

Or is there something else I miss here?

[mskarbek]

I'm working on wrapper around libzfs that will expose D-Bus API, is that will solve licensing problems?

[vpodzime]

I'm working on wrapper around libzfs that will expose D-Bus API, is that will solve licensing problems?

Yes, using a DBus API in our GPL-licensed code is perfectly OK no matter what provides such an API.

[ghost]

@dreamcat4 commented on 28 Jun:

According to Richard Stallman, who created this whole unnecessary legal mess in the first place.

That seems an unreasonable thing to say. Here are the facts, as I understand them:

• Stallman was not responsible for the terms of the CDDL. The CDDL was created by Sun Microsystems in the expectation that it would prove incompatible with the GPL.^{0, 1, 2}
• Stallman was not responsible for the decision by Sun Microsystems to release OpenSolaris (and hence ZFS) under the CDDL.⁰ (IMO, this decision was a mixed blessing: it was good for humanity for Sun Microsystems to publish ZFS as free software, but their decision to publish it under the CDDL alone can only realistically be seen as an attempt to prevent or postpone its integration with GNU/Linux. After all, when Sun wanted cross-platform adoption for other products, such as Java, it was willing to publish them under the GPL.)

With that out of the way: thank you to you (and to everyone else) in this thread for your enthusiasm to find a way to support ZFS within udisks. I'm sure you will succeed, even if it will mean creating a shim/etc that cannot be distributed directly within udisks. As long as it ends up being done legally, I look forward to it!

[ghost]

@kpande wrote:

that you just posted is FUD.

Are you saying that Stallman was responsible for the terms of the CDDL, or that Stallman was responsible for the decision by Sun Microsystems to release OpenSolaris under the CDDL? I'd like to see your references.

[ghost]

@kpande wrote:

i'm saying that merely providing optional support for libzfs does not constitute a violation of GPL or CDDL, as the version of CDDL used in ZFS allows its source code to be licensed under GPL. it is in binary form that these conflict, as both require binaries be distributed in the original license.

this is how distributions would work around the perceived licensing issues; by merely not building the support and distributing an allegedly offending binary.

How does that make what I wrote FUD?

[ghost]

@kpande wrote:

oh, you presented speculation as fact.

Where?

[rlaager]

The whole topic of linking and derivative works is complex. Debating it tends to be unproductive in practice. GRUB, whose copyright is held by the FSF, takes the position that linking to libzfs is okay on Solaris (via the system library exception), but not elsewhere (e.g. Linux). On Linux, they shell out to the CLI tools, which have switches to help make the outputs parseable. That approach (shelling out to the CLI tools) is probably the best answer here, especially in the short term.

[FireDrunk]

Problem with ZFS is that the shell output may not be "API" Compatible, but LibZFS should be. Perhaps we can get answers from the ZFS team on how "stable" the JSON output will be, and do a shell out to the json output of CLI tools (which is probably best anyway).

I'll ask on the ZFS Github for some comment.

[dreamcat4]

My same recommendations still stand. Nothing has really changed here.

@sampablokuper you raised a valid concern / point about the Sun CDDL being suspicious. Researching this issue actually caused me to look very deeply at both licenses and the history behind / around each of them. In my examination I was also initially at the starting position that the CDDL was probably a bad license... and because of the same general reason you cited: it being created and/or chosen by a corporation (Sun Microsystems in this case). I simply could not believe that Sun (even back then) would permit their engineers write their own license terms, and without it getting in the way of some vested corporate interest. So you know, I already looked into these things from that from that angle too. Clearly you are still on that journey. So I wish you good luck in finding the truth behind that! But like the other people said here: going any further into those histor(ies) doesnt actually move this specific issue forwards.

Everything else is already in my previous comments.

[rlaager]

I'm not aware of any JSON output format having been merged, and the PR seems to have been abandoned and closed.

The CLI output, especially with -H and -p as appropriate, is quite parseable and plenty stable. Many shell-script tools use the zfs output. GRUB uses the zpool output, so if that changes too dramatically, booting from ZFS would break, so there is incentive to not break that.

[Alexey-I]

Json support has been merged into zfsonlinux:master: https://github.com/zfsonlinux/zfs/pull/7281

[CarlGill]

Was there any movement on the CLI based implementation? Or did the everyone only come here to comment on licensing issues and then disappear :)

Thanks to everyone for their time spent on this.

[vpodzime]

AFAIK, there has been no progress with any actual implementation so far. But judging by the couple latest comments I think the work on a CLI-JSON-based support can start. And I think it should be done the way other things are supported in UDisks2 now -- support for stateless operations/manipulations in libblockdev and statefull objects with DBus API in UDisks2.

However, what also hasn't changed, AFAIK, is the lack of interest in ZFS among the core developers of both libblockdev and UDisks2. A good starting point for anybody who wants to contribute actual code would be https://github.com/storaged-project/libblockdev/blob/master/README.DEVEL.rst

[Redsandro]

What causes a ZFS pool to show up in the nautilus/nemo 'Devices' list if ZFS isn't supported by udisks?

[FireDrunk]

Probably a zdev.

[vojtechtrefny]

It is a block device with a (mountable) filesystem visible by udev so we "support" it, but we can only show information that udev already knows about the device. This issue is about support for "managing" ZFS.

[Redsandro]

@vojtechtrefny clear, thanks.

[brando56894]

It looks like this has been stagnant for about a year, any development on this? I'm currently using Ubuntu with ZoL, and while managing ZFS via the CLI is fine, I'm a fan of pretty GUIs. I was using FreeNAS but Bhyve sucks for virtualization compared to KVM, and I can't pass through my drives to a FreeNAS VM because I updated my feature flags, so it's now ahead of the FreeBSD modules.

If this is dead in the water, I guess I'll have to wait until FreeBSD switches over to ZoL in FreeBSD 12 to get a nice GUI.

[tbzatek]

It looks like this has been stagnant for about a year, any development on this?

Things have been status quo since last @vpodzime's comment. The idea is not completely abandoned and we would welcome contribution to libblockdev and a udisks module built on top of it (and ideally a maintainer taking good care of that code).

As far as licensing is concerned, my take is that loading shared objects via dlopen() is analogous to compile-time dynamic linking. It's just about the moment when dynamic linker resolves symbols, however the foreign binary code is still being pulled into the single process and the licensing terms apply without difference.

[vojtechtrefny]

As far as licensing is concerned, my take is that loading shared objects via dlopen() is analogous to compile-time dynamic linking.

There is no difference between linking and dlopen for licensing. Any implementation definitely needs to use the command line tool to avoid licensing issues.