DemiMarie
commented
2 years ago Is GitHub sponsors an option?
Open jittygitty opened 2 years ago
DemiMarie
commented
2 years ago Is GitHub sponsors an option?
jittygitty
commented
2 years ago Hi @DemiMarie Thanks I didn't know about "Github Sponsors", but it looks like it's only "Donation" based like Patreon etc?
Donation sites would need OpenZFS leaders to setup one "official" account with them, or for individual "code contributors" to have their own individual accounts.
This poll is mainly for Bounty/Rewards type platforms where any of us can post our own Rewards/Bounties for a "Specific Issue" and Bounty-Hunters/Coders can compete and claim reward/bounty if they Solve Issue or Implement the Feature Request.
Ornias1993
commented
2 years ago I think money isn't the primary blocker for ZFS to be frankfully honest.
Having worked on the ZSTD implementation with @c0d3z3r0 and @BrainSlayer I can honestly say that elitism, favourism and a complete dis-interest within the maintainers and frequent-contributors clique, has caused more delay than all mistakes and bugs encountered in our code combined.
And we where doing that for free (me testing, them withing code and c0dezero restructuering). Giving us money for said feature, wouldn't have changed the fact we where mostly waiting for months and months on end for ANY progress on review, feedback and merge.
It was only when Allan Jude got back involved (which is within the maintainers clique of frequent contributors), that "magically" we got attention from maintainers again.
To be fair: I might be very harsh and passive agressive a lot of the time. But I rather have an productive ass (for free or for a bounty) than totally unresponsive people with a superiority clique.
Then there is the problem of users not actually finishing Proof of Concepts, they get some speaking time on a conference for self promotion and quickly afterwards stop working on their PoC of a new feature. This would also not magically solved by adding bounties. (often the free promotion these people get is worth significantly more than any bounty they would recieve for the hardest part: finishing a project)
Besides these things, there is also the subject of some OS's getting more attention than others, by how currently big sponsors are a very small group of companies. Often related to the mainainers and big contributors, which is a quite clear conflict of interest
What I want to highlight with this, is that the lack of bounties is not the biggest organisational problem that is holding back important/big project features.
jittygitty
commented
2 years ago @Ornias1993 I appreciate your feedback and experience and especially thanks for any contributions like with ZSTD you mentioned. I realize having bounties may not be able to fix some of the hurdles you mentioned but I can't help but feel that they could help "indirectly". The main thing that worries me is the possibility there isn't anyone able/willing to jump on even a well funded bounty because when I look at some of the C file system code on here, it looks just as "baffling" to me as assembly. So my worry is that most of society has moved towards wasting time on social media and not much interest in heavy brain work and it may be extremely difficult to find anyone unfamiliar with zfs to come in and code at this level. My hat goes off to all the code contributors on here, I could only dream to be so proficient. I just hope that Crowd Funding issues/features 'might' have a chance to lure in some coders. And even for existing gurus, who knows maybe when they see a feature gaining excitement that's confirmed by the dollar value of attached reward, who knows maybe it might get them to work on it. Lol and who says you can't put a Bounty on getting a "Review" for a pull-request : D
Now I wish I could find some Insert POLL button on here so people could easily vote on preferred Bounty Platform...
Ornias1993
commented
2 years ago The problem I tried to highlight was that even if you have a bounty, people are going to drop out because of the extreme lack of activity/interest in the maintainers for PR's made by people outside their clique.
jittygitty
commented
2 years ago @Ornias1993 That may be true, but in life I always try my best to give everyone the benefit of the doubt, I actually wrote a post asking something of a "clarification" about this project but I didn't want anyone to be offended by the question, so I shelved it. It was simply me wanting clarification if this was a traditional community project or a "community Version" of the product of a company or group of companies, because there is a difference. The question came with a request as well. Anyway if you look at the issues I created, I think you will realize that I am doing my best to find out if the maintainers will be open to PRs of a certain kind. I want to give everyone the benefit of the doubt, and perhaps if I am very persistent in my requests but at the same time try to be respectful and considerate, since I really do greatly appreciate the work of all the leaders and contributors, that things will work out well and we can start making better progress and gaining new users, and if not, and I find out the companies or other interests must control the direction and not the community, then at least I can start a new fork of the project while still being on good relations and able to work together on certain zfs issues when goals align and separately on other issues. If I find out the only way for entire community to have more influence is to fork, I had planned to, but I "really" wan to completely avoid that, if its possible to avoid it. If that turns out to be the only way, I had thought of "FreeZFS" but domains were taken etc.
My goal is simple:
So if it will be impossible to have PRs even accepted in a "BRANCH" that abide by the two above, then I guess will be no choice.
I plan to ask a question in the zfs FIEMAP implementation issues that have been stalled, to ask if the work they had done was avoiding any GPL-Only exports such as iomap-fiemap and if work could be completed easier if they simply use every system call without caring if it is gpl-only or not. If you don't distribute, or don't distribute as binaries, there's no legal reason why you can't give your end user a "script" that tells ZFS to put on a GPL suite when building/compiling and so have access to all gpl-only system calls ioctl's etc, or your build strip could strip gpl-only from everything in Linux, all this is perfectly right of user.
The ONLY reason that someone might object to this, is nothing to do with license, it is only if they are distributing/selling "APPLIANCES" whether physical hardware or pre-built VMs etc that come pre-installed with BINARIES. Obviously that would be "Distribution" and then and only then the "License" issues come into play. But if "building" from source there's no license issue.
I have only recently learned of some possible GPL-only system calls that "maybe" could have impeded some features if trying to abide by them. I had asked for simple "DOCUMENTATION" how to build ZFS with native support for fpu SIMD without the workarounds but nobody provided any, see https://github.com/openzfs/zfs/issues/11357
I was very thankful to @behlendorf because he did not shoot down my issue, but to the contrary he seemed supportive, unless I'm mistaken and the actions in labeling that his username took were automatic and he didn't actually look at it himself etc?
So I was encouraged by that as a sign that it will be possible to better leverage already included Linux code towards faster implementation of Reflinks + Offline-dedupe and other features that can leverage Linux Fiemap and others. This is the reason for my efforts in trying to get the community to agree on a Crowd Funding Platform, because all I wanted is for the leaders to accept us and our goals, I don't expect them to work on these things "themselves" if its not in their interest, if they must for example "Distribute Binaries" in their products so they are "Unable" to use gpl-only exports in their products, or if they are using distributions like BSD or others based on opensolaris/illumos kernels (that lack Fiemap and others that Linux does have).
Anyway I hope I have clarified the intent here well enough and others are open to the idea of joining together to try to lure in some additional talent that we can fund via Bounties/Rewards for solving issues or implementing features etc and implementing them by following the step (1) and (2) guidelines I outlined above. I don't expect the leaders here to participate in this way if its in any way not aligned with their own interests, I will be very happy if they simply accept our interests also and allow us to at least have PR's that follow (1) and (2) to be revied and accepted at least into a "BRANCH" of openzfs repo etc.
Apologies for long post! Now back to what Bounty Platform should we use? :)
mskarbek
commented
2 years ago @jittygitty nobody sane will be using kernels with GPL exports replaced with non-GPL one, this can be feasible for home or experimental usage at most. You can't ask for an effectively useless implementation of a feature. I can't, and I will not be replacing RHEL kernel with custom one, and I'm not the only one with such constraint. I am happy for you that you could YOLO into production whatever you want, but believe me you are in the minority.
rincebrain
commented
2 years ago Just ignoring EXPORT_SYMBOL_GPL doesn't buy you much for additional features, for OpenZFS - the reasons that something like BRT is difficult on ZFS are architectural in ZFS, not a Linux export problem, for example.
I would suspect, not having asked anyone, that a reason nobody chimed in to the FPU question was that nobody has any desire to encourage people to go patch Linux to circumvent things, then either report bugs that aren't reproducible without such changes or have other people point to it as a documented solution, even with a disclaimer in big letters THIS IS UNSUPPORTED DO NOT FILE BUGS AND YOU GET TO KEEP ALL THE PIECES, and take it as endorsement. (Personally, it's also just not a thing I've tried.)
Not working for anyone who pays me to play with ZFS, my perception has been that difficulty getting things reviewed is mostly a paucity of available time, and that people can get people who work with them during the day to review things is a side effect of "oh, I reviewed this at $DAYJOB and we've been running it for 6 months, this will be fast". But it's certainly the case that everyone has trouble getting things reviewed in a timely fashion sometimes, or else the open monthly meeting wouldn't have a section every time for "can some people please look at these PRs, they've been waiting for a long time" (which, I should be clear, is not just where people link their own).
I can't claim there is or isn't a secret club for more consistent turnaround time on PRs and lower friction - if there is, I've not gotten an invite. But I've not personally seen significantly more friction for getting things merged than I would expect from difficulty wrangling humans to read unfamiliar code, so far.
As far as bounties on features, my $0.02 is I don't know that it would be a net gain - if it incentivized new or existing contributors to work on things, great, but I think many of the existing contributors are limited by total productive time available in the day, not being paid differently for it, and money has a way of distorting incentives pretty phenomenally - if you only get paid if your thing gets merged, are you likely to report if you discover that it burns down pools on the second Tuesday of the month? (And if you only get paid if all bugs get fixed within an SLO, why would you have any reason to contribute anything nontrivial at all?)
(Not that contributions from independent unpaid contributors or companies don't have similar fallout sometimes, but they're not often directly incentivized to hide issues, since usually they wrote something intending to use it.)
Ornias1993
commented
2 years ago In general that's indeed the point: The lack of progress is not reallyfinancial issue, at least: not that we've heard much here.
There are lot of reasons to contribute or not-to-contribute, finances are not often the primary issue on OpenZFS that I heard. In important big projects where it can become a problem, there are often solutions found with some form of direct sponsorship of a certain feature by a certain company. But that's a very minor amount of issues.
jittygitty
commented
2 years ago @mskarbek @rincebrain I appreciate both of your inputs also. But you have misunderstood what I've said and its implications.
I will try to explain how what I have proposed entails a "whole lot less" than what tens of thousands of companies are already doing in their own products, and I will give two examples, one dealing with distribution, and the other dealing with "SaaS".
You might be surprised perhaps by what I tell you if you think that "no one" is interested in running custom kernels in production. Have you ever heard of MontaVista Linux? They sell/ship/distribute customized Linux without even providing the sources to anyone but the direct person who buys their "custom linux. The code they add is in that sense proprietary and custom, you can't download it anywhere, please do show me a link where to download MontaVista Linux. Millions of devices are then shipped with that MontaVista Linux further "mangeld/customized" and you generally "never" see the sources even. Let me tell you that many HP DesignJet Printers/Plotters ran MontaVista Linux, customized further by HP yet no code anywhere. Now I'm not condoning the distribution of hardware with GPL without providing the source, but I've looked "everywhere" and could not find HP providing sources or even a page where you could request it. My point here was more about the fact that MontaVista Linux was sold as "source" only and "legally" they are not required to provide source to anyone other than buyer.
The other group of companies that make use of very highly customized software and linux kernels are those SaaS providers, ie Software as a Service, and they do, because they "can" intermingle" closed-source proprietary and all sort of incompatible license if they were distributed, they intermingle them on their platforms that run their "Software ass a Service" systems etc.
Just the usages that fall in only these two types of examples prove that customized kernels are quite common, there's many more I can give about hypervisors etc but already post getting too long. And my "MAJOR" point I wanted to point out is this:
What I have proposed is a LOT LESS! I am proposing two build options which do not require any real "customization".
In fact if you read through pages of details FSF GPL legal jargon you'll find out that from their perspective when another module binary talks to GPL kernel closely that it becomes GPL and that technically you could ship/distribute it even as binary but the catch-22 is you would have problems complying with providing the sources to go along with those binaries since you can't change the cddl license on "distribution". So that's the only catch-ya. Not a big deal for tens of thousands of companies which do much deeper customizations as I have mentioned above. And this option does "literally ZERO" code customization.
OR
(Of course no repository would contain either ZFS modified like this, nor would any repository contain Linux modified like this, there simply will be SCRIPTS that a user can "choose" to run on their systems if they should choose to do either (1) and/or (2))
@tonyhutter You or others knowledgeable I'm sure can easily verify there's no legal issue at all with what I proposed above. The GPL clearly acknowledges that the License "only" comes into play during distribution and is meaningless to end user in private.
So you see neither 1 nor 2 would in any "practical sense" customize the linux kernel, all you do is ALLOW ZFS to use some of the system calls and code inside Linux that ZFS would otherwise not have any access to. Someone gave one example of the Linux page cache that because of gpl-only export, ZFS can't use and has to do its own thereby using more RAM than it would have to use if ZFS could have access to Linux page cache like other (gpl) software can.
I want to run ZFS on Linux in best way possible. If there's no legal reason why we can't then why shouldn't we be able to use all the Linux kernel system calls and APIs to the fullest so ZFS runs as well as possible? Why should we be forced to use buggy less than ideal "workarounds" for fpu SIMD etc, for what great reason? They were there before, they were taken away quite recently by the Linux kernel developers, I'm sure that makes Oracle very very happy, especially if no one des what I proposed, since a more feature-full ZFS on Linux is not good news for their business just as mysql was not good news for their business.
So there seems to be a lot of confusion about licenses which I'm hoping the above can help clear up.
You know when I first read this in a long 7-page technical discussion about ZFS at Phoronix, I thought this guy had it all wrong when he said: https://www.phoronix.com/forums/forum/software/linux-gaming/1269446-proposed-reflink-support-would-provide-big-space-savings-for-wine/page7
"...There is a completely different problem here. ZoL developers don't want to say what the problem is. Hooking up Linux kernel reflink interfaces that are under GPLv2 only to CDDL only ZFS deduplication code where the ZFS reflink code is happened to be very legally problematic. Telling users to use other features or there is no gains to the reflink feature means they don't have to say they are license stuck."
"ZFS by design has reflink. If file system driver exposes that to user space that a different matter. Of course when you understand the license problem here you understand why the ZoL developers are attempting to weasel their out of having to implement reflink as userspace usable feature even that they are using reflink feature internally."
I was hoping someone in leadership would clarify/confirm that he is wrong, because I thought the likely reason was just that the majority of active leaders were simply using more BSD or illumos based distro's versus Linux. But now I don't know, perhaps they are also "Distributing" Hardware or VMs "Appliances" with pre-built BINARIES and so for their use case they are "stuck" and can not make use of "gpl-only" Linux Export Symbols?? If that is the case then that guy was not completely wrong after all.
It would be quite easy for those in leadership to clear things up themselves, all we can do is theorize since we do not know their work/business models etc and if/how to what extent they are impacted by Linux Kernel developers starting to CHANGE more and more free-for-all "Export_Symbol" to only "Export_Symbol_GPL", and I've heard they've even started doing this to OLDER kernels backporting this draconian "lock-down" to kernels that all these years were not locked to gpl-only Exports etc.
So again, what I have proposed is completely legal no questions or gray area about it, and there's no real technical complication at all since you really are not truly CUSTOMIZING anything, to use an analogy we would simply be UNBLOCKING some Communication Ports in a Firewall so ZFS can TALK to LINUX, that Linux developers started to "close" more and more.
And again to be very clear, I'm not asking that my option 1 and 2 to be the only way forward or primary way here, I simply want to know if those of us who want what I proposed if it will be "ACCEPTED here as an option" that's all! If not, its ok we'll go away.
(Sorry for any mistakes I typed this fast in a hurry, will be happy to clarify or make corrections.)
rincebrain
commented
2 years ago First, you can't mingle pools between Oracle ZFS and OpenZFS - even if you went with pool/FS version v28/v5, and even if their cp --reflink is just built on an unmodified dedup (which is I believe false, as one of their announced changes in a newer pool version is "dedup v2"), Oracle made breaking ondisk format changes without requiring revving the version, so it will fail in strange ways.
In addition, even if it were an option to just do cp --reflink with the existing dedup implementation, I think people would veto it. Dedup has too many sharp edges and negative performance implications, which is one reason Pawel spent a lot of effort coming up with a much lighter-weight solution with fewer downsides.
Finally, the reason it's not been done is not some conspiracy about Linux blocking ZFS - it's because it's a hard problem.
As far as I can tell, nothing about reflink is EXPORT_SYMBOL_GPL-guarded - vfs_clone_file_range et al are EXPORT_SYMBOL, and I don't know of any way currently to guard the function signature for ops functions in a struct. (That said, even if it were, great, go implement a custom call like pjd did for testing before the syscalls are hooked up and people can use zfs_cp or whatever. Gross, but the functionality is desirable enough that I think if it were necessary people would put up with it.)
jittygitty
commented
2 years ago @rincebrain This has nothing to do with Oracle ZFS at all, I never even used it, I don't care about Oracle ZFS. My issue at https://github.com/openzfs/zfs/issues/13349 if you read it will clear things up I think. The point is that if we leverage existing Linux code such as FIEMAP, we can get a lot things done on Linux, easier and faster than on BSD and illumos because those do not have Fiemap and existing reflink code accessible already. So it has nothing to do with Oracle ZFS at all, we're talking Linux.
Iomap-Fiemap is GPL-Only I only very recently found out. Previously I had thought nothing in FIEMAP was gpl-only so I was like hey what's the holdup, why is that FIEMAP implementation on here stalled and deserted...but recently I found more gpl-only exports that "might" need workarounds (IF you don't simply go by one of my (1) or (2) options that I presented above, if you're ok with (1) or (2) above its much easier), and I was going to and will, post in FIEMAP issue to ask if that was part of slow down.
But exactly, thanks for acknowledging that this functionality of cp --reflink and Offline-Deduplication is so desirable for many of us that have been begging for it for like the last twelve years, so I was just hoping we're not forced to fork to do it via (1) or (2).
And many "other" features can very likely gain much benefit from (1) and (2) as well, not just reflinks and offline-deduplication.
rincebrain
commented
2 years ago You linked to a long rant about how ZFS secretly could do this tomorrow if it weren't for that meddling Linux and its pesky EXPORT_SYMBOL_GPL, which was grossly misinformed.
Nobody involved has claimed it's not useful functionality. The claim is that it's nontrivial and that Linux restricting any particular interface is not even visible on the scale of "things blocking reflink being implemented".
(And lest you think otherwise, I don't speak for the project in any way - I'm only listed as a member of the org because I asked for the ability to tag bugs with things like Encryption.)
jittygitty
commented
2 years ago @rincebrain The "rant at Phoronix" thread that I linked to I already pointed out when I posted it that I did not believe/agree with the guy, I simply thought the reason was that the leadership here had more interest in non-Linux distributions, not that they were trying to "hide" that they were impeded by Linux guys closing more and more export_symbols and making them gpl-only, but since you say you'r'e not leadership per se, the leadership can surely clear things up easily themselves on that claim.
And I never said that removing the barrier of gpl-only exports would make reflinks and offline-deduplication and other features a complete peace of cake. I was not even as optimistic as @zenaan although I did agree with him on the implementation path. You can see his comments at #10552 and https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tfa22fbf65c5411f0
My point is that it seems to make things a whole lot easier for some features, and in #13349 I presented plenty technical reasons why, in various linked PDFs etc and patch-sets etc and asked anyone disagreeing to please give technical reasons.
Again for the leadership this a question of inclusion, will we be tolerated/allowed to exist even as second class citizens in a BRANCH of openzfs for features we wish to implement by following (1) or (2) above, or will we be forced to leave and fork?
Anyway I've explained the benefits of my proposal and why its not problematic for others not wishing to use it, that they can continue as before, and that for those of us who choose the (1) or (2) method it can be a very helpful for us for some features.
And by the way, the original intent of this ticket, to VOTE on a BOUNTY/Rewards platform for us to use, was for the benefit of all type of coders/contributors, not just for those who will implement via my proposed (1) or (2) method but even for those who need to "Distribute" hardware or software appliances or binaries etc. So differences in ideology on (1) or (2), pro or con shouldn't really impede us in any way on deciding on some BOUNTY/Rewards platforms to use etc. Obviously those coders who go by (1) and (2) may have, for some specific features, a much easier time implementing them, then those who must implement the same feature without being able to use certain gpl-only exports, so in that case yes some would have it easier.
So I'm not asking anyone to do things my way, everyone can put their mouth and money wherever they want that suits them. The only thing I'm asking is if the way I proposed, ie (1) or (2) can be accepted even as second class citizens in a "Branch" here.
Thanks to everyone for their input, so now back to Funding/Bounty Platforms, any interest in funding through them or not?
sempervictus
commented
2 years ago The slow response from maintainers is definitely a huge issue because spot-contribution (especially of big features) is almost universally a sprint function in FOSS projects whereas the operational workflow is reminiscent of the bureaucratic processes of the grant-based sciences world. The fact that we only now have an official decision to break existing functionality by implementing new compressor functions over old ones is indicative - the epic effort @Ornias1993, @BrainSlayer, and @c0d3z3r0 were putting in to get current-revision ZSTD into ZFS had stopped before that concern was handled by management (and without addressing the architectural direction, there could have been no decision made on how to implement their code). A loss for the entire community, and hopefully those folks will someday make a new PR against master with 1.5.3 or what-not (because @BrainSlayer's master branch is a bit hard to track neatly for just the ZSTD commits). This, IMO, is why ZFS is falling behind commercial efforts so bloody fast - they have customers driving demand with huge performance requirements (latency and throughput) with advanced data functionality (now matching ZFS') while we have stale PRs on the vine and backported stable tags not representative of the R&D going on because master is basically considered unstable relative to the tags (further delaying the applicability of the code which does actually get merged for users). ZFS intended to avoid the bitrot problem for data, IMO, we need to be concerned about bitrot for code (if not functionally then in comparison with other storage solutions out there). If all big datasets end up on proprietary storage, then "someone" will effectively own everyone.
To the initial inquiry about which funding mechanism to use - its fairly irrelevant, short of how much is taken off the top by the middleman and which tax haven the thing inhabits. Things like the async DMU work will likely cost many thousands of dollars to hire someone qualified to spend that much time on it - significant tax implications there. We might as well just set up an escrow account somewhere and have actual contract terms with specs for completion of tasks since we're talking money changing hands. A good project manager might be the first goal-post here - someone experienced with complex projects atop FOSS code in socially complex ecosystems, preferably with some relevant language skills...
BrainSlayer
commented
2 years ago what means efforts stopped. i constantly maintain it with latest versions and is in sync with zfs master. so its relativily easy to track. i also squashed all my current commits. so that its even more easy to review
sempervictus
commented
2 years ago @BrainSlayer - thanks, i meant the pull requests for getting it merged into OpenZFS proper. Also much appreciate the squash - will try to cut an hour or two this week to refresh off your code again :). Since maintainers have decided that implementing new compressors at the cost of breaking dedup is the path forward, it might be viable to get iterations off your master branch for (at least functionally relevant) ZSTD tags to get upstream.
sempervictus
commented
2 years ago @jittygitty - the GPL fix is easy, if you don't intend to redist, just change the license tags from CDDL to GPL:
grep -r CDDL include/zfs/|grep -v '\*'|grep -v bsd|cut -d':' -f1|while read FL ; do
sed -i 's|ZFS_META_LICENSE = CDDL|ZFS_META_LICENSE = GPL|; s|#define ZFS_META_LICENSE "CDDL"|#define ZFS_META_LICENSE "GPL"|' $FL
doneIn either case (messing w/ upstream kernel code or this hack) you're violating someone's license arrangement though, so i doubt you'll get too far with this approach as nobody wants to get sued by patent trolls (either the GPL ones or Oracle/IBM). The GPL-only exports are capricious, added mid-stream to stable releases, and seemingly intended to confound us and other out-of-tree projects to favor the in-tree competitors such as the bit-rot filesystem. If OpenZFS is to be anything but an archival store in the future, it needs to work with the paradigm at-hand: the code and binary product have to work in the ecosystems for which they are destined; so in Linux-land, we have to play their game and pretend like we agree with their delusional utopian licensing scheme.
BrainSlayer
commented
2 years ago @BrainSlayer - thanks, i meant the pull requests for getting it merged into OpenZFS proper. Also much appreciate the squash - will try to cut an hour or two this week to refresh off your code again :). Since maintainers have decided that implementing new compressors at the cost of breaking dedup is the path forward, it might be viable to get iterations off your master branch for (at least functionally relevant) ZSTD tags to get upstream.
whats means breaking. i still run the code productive with dedup. of course dedup is a problem if a new block is written. but it solves itself when the older block is rewritten too :-) i have indeed a script which rewrites all data on the filesystem for such purpose. this so called "breaking" is no blocking point. its harmless. but we might get significant better performance with newer versions.
sempervictus
commented
2 years ago @BrainSlayer - the DDT entries made by older compressors may not match the newer entries, resulting in duplication of data blocks which would otherwise have been deduplicated. So if you had a bit or two changed in the compressed data output of ZSTD between versions, the DDT-stored-checksum will differ and create new data instead of marking a reference on the existing block.
I mean "breaking" in the sense that ZFS offers a set of functions, including dedup, which are no longer consistent over versions of ZFS or even across different operating systems with the same version of ZFS (gzip on BSD and Linux will produce different outputs). Its an architectural defect, not specific to the implementation of ZSTD in ZFS - we'd need to store uncompressed sums, and then the only way to verify them would be to decompress the data a bunch in the pipeline, which is a fair deal of overhead. For enterprise users who may have 100T of logical data, deduplicated down to 20T of on-disk, on a store with only 30T of capacity, and snapshots mandated for retention by legal constructs; their storage system will flood itself if they re-write a significant portion of the already deduplicated data without deleting those snapshots because they expect from the stated feature-set that deduplication works as advertised. AFAIK it was between this and restructuring how we store/process checksums, and its easier/faster to just introduce breaking changes for dedup since dedup is already broken by the fact that Linux uses the ZFS compression code but BSD (and others?) do not.
BrainSlayer
commented
2 years ago @BrainSlayer - the DDT entries made by older compressors may not match the newer entries, resulting in duplication of data blocks which would otherwise have been deduplicated. So if you had a bit or two changed in the compressed data output of ZSTD between versions, the DDT-stored-checksum will differ and create new data instead of marking a reference on the existing block.
I mean "breaking" in the sense that ZFS offers a set of functions, including dedup, which are no longer consistent over versions of ZFS or even across different operating systems with the same version of ZFS (gzip on BSD and Linux will produce different outputs). Its an architectural defect, not specific to the implementation of ZSTD in ZFS - we'd need to store uncompressed sums, and then the only way to verify them would be to decompress the data a bunch in the pipeline, which is a fair deal of overhead. For enterprise users who may have 100T of logical data, deduplicated down to 20T of on-disk, on a store with only 30T of capacity, and snapshots mandated for retention by legal constructs; their storage system will flood itself if they re-write a significant portion of the already deduplicated data without deleting those snapshots because they expect from the stated feature-set that deduplication works as advertised. AFAIK it was between this and restructuring how we store/process checksums, and its easier/faster to just introduce breaking changes for dedup since dedup is already broken by the fact that Linux uses the ZFS compression code but BSD (and others?) do not.
i know. but its harmless. its just not deduplicated under certain circumstances. and if you rewrite the previous block its deduplicated again. but i wrote this again. and its can be easily solved by rewriting older blocks. of course this should not be done automaticly. i just do it by rewriting all files with a simply script. storing uncompressed sums on the other side is no solution. it will decrease performance since a uncompressed sum is done on a larger block. but would never say its a defect. it will not break the fs function and will not cause damage. a upgrade script is a better approach. for simply updating all written blocks. this can also be done in a background automatism. ever raid is doing this if you replace the hard drive.
DemiMarie
commented
2 years ago @sempervictus I think ZFS needs to have a legal entity behind it that can handle the financial and management matters. @oshogbo Is the FreeBSD Foundation a good choice? Or is there an illumos foundation?
Ornias1993
commented
2 years ago @sempervictus I think ZFS needs to have a legal entity behind it that can handle the financial and management matters.
To be fair it's completely weird that there isn't already
@oshogbo Is the FreeBSD Foundation a good choice? Or is there an illumos foundation?
The FreeBSD foundation is even more close-knit than OpenZFS leadership, does not respond to formal complaints and is extremely unopen about their sponsorship activities.
Illumos is on life-support, but is also a minor player on the ZFS scene anno 2022 (mostly legacy users)
rincebrain
commented
2 years ago @BrainSlayer - thanks, i meant the pull requests for getting it merged into OpenZFS proper. Also much appreciate the squash - will try to cut an hour or two this week to refresh off your code again :). Since maintainers have decided that implementing new compressors at the cost of breaking dedup is the path forward, it might be viable to get iterations off your master branch for (at least functionally relevant) ZSTD tags to get upstream.
I have a branch with this implemented, and with versioning it so it doesn't break any of the edge cases that expect compress(decompress(X)) == X, too.
I just didn't move forward with opening a PR because the gains were minimal to negative on at least the compressor side, and the additional complexity didn't seem like a win - the same reason I didn't open one for the LZ4 compressor, for that matter.
I'd probably argue against someone else's PR doing the same thing for the same reason, unless they have counter-examples.
jittygitty
commented
2 years ago @sempervictus Thanks I appreciate all your thoughtful input also, but I need to clear up some confusion again license-wise, I am quite familiar with such license issues and I can provide more context and links to prove there's like "zero" patent troll risk.
But maybe I should do a thorough license explanation in a separate thread/issue to ease everyone's concerns? This issue was mainly to vote on if we can get on board with funding some issues via Bounty/Rewards and if so which platform/s to choose.
@DemiMarie @Ornias1993 Although some official "management" for such matters would help, its not really needed at all for merely the things I have proposed in terms of Bounty/Rewards funding. Where such "OpenZFS Management Team" would come into play is creating an "Official Account" where blanket/General "DONATIONS" could be sent on sites like Kickstarter or "Github Sponsors" or "Patreon" etc. But of course, INDIVIDUAL Contributors can setup their OWN separate accounts on such "Donation"-based platforms where we could perhaps easily CLICK to DONATE to whatever coder/contributor we want to fund.
The "Difference" with this issue for voting on "BOUNTY/Rewards" is that its a bit more similar to a CONTEST like on 99designs.com which is for "graphics". But the point is that all of us can sign up with our own individual accounts to a "BOUNTY" platform like one of the ones I mentioned at the top in this issue, and we would identify as part of "OpenZFS".
The reason to try and organize ourselves a bit and decide on the actual platform or platforms to use is so we can coordinate to link ourselves as part of the "same team" on there so our funding can appear "pooled" instead a little bit scattered everywhere.
By identifying our individual accounts as part of OpenZFS this means that the platform can tally up, add up the TOTAL of all our individual funding amounts so that a coder-"Bounty Hunter" searching for TEAMS to work for, will see, wow cool, there's $20K up for grabs put up by the OpenZFS "TEAM". So you see we can do all this without any "help from anyone in leadership" here.
So each one of us can put up some money in ESCROW on these platforms that are tied to SPECIFIC issues. If ten of us fund the same feature and that feature ends up with $10K in Escrow waiting for a Coder/Bounty-Hunter to "implement" and win $10K, I think it increases the chance that a couple coders may give it a shot and try to implement and win the $10K. If one guy wins, the other who lost would have learned a lot of OpenZFS code in the process and would likely jump on another ZFS Bounty etc.
That is why I see such BOUNTY/Rewards type funding as 'potentially' very beneficial in short-term and long-term. Anytime you lure people in to get familiar with your large code tree/repo, I think its a good thing. At some point some may continue such work without getting paid bounties, once they learn the code and maybe start using and enjoying zfs for themselves. And for all of us, because our funds are "recoverable" and only get paid out if a working solution is provided, its a "safe" investment.
sempervictus
commented
2 years ago @BrainSlayer - the DDT entries made by older compressors may not match the newer entries, resulting in duplication of data blocks which would otherwise have been deduplicated. So if you had a bit or two changed in the compressed data output of ZSTD between versions, the DDT-stored-checksum will differ and create new data instead of marking a reference on the existing block. I mean "breaking" in the sense that ZFS offers a set of functions, including dedup, which are no longer consistent over versions of ZFS or even across different operating systems with the same version of ZFS (gzip on BSD and Linux will produce different outputs). Its an architectural defect, not specific to the implementation of ZSTD in ZFS - we'd need to store uncompressed sums, and then the only way to verify them would be to decompress the data a bunch in the pipeline, which is a fair deal of overhead. For enterprise users who may have 100T of logical data, deduplicated down to 20T of on-disk, on a store with only 30T of capacity, and snapshots mandated for retention by legal constructs; their storage system will flood itself if they re-write a significant portion of the already deduplicated data without deleting those snapshots because they expect from the stated feature-set that deduplication works as advertised. AFAIK it was between this and restructuring how we store/process checksums, and its easier/faster to just introduce breaking changes for dedup since dedup is already broken by the fact that Linux uses the ZFS compression code but BSD (and others?) do not.
i know. but its harmless. its just not deduplicated under certain circumstances. and if you rewrite the previous block its deduplicated again. but i wrote this again. and its can be easily solved by rewriting older blocks. of course this should not be done automaticly. i just do it by rewriting all files with a simply script. storing uncompressed sums on the other side is no solution. it will decrease performance since a uncompressed sum is done on a larger block. but would never say its a defect. it will not break the fs function and will not cause damage. a upgrade script is a better approach. for simply updating all written blocks. this can also be done in a background automatism. ever raid is doing this if you replace the hard drive.
Oh yeah, if we have the ability to rewrite the data on-disk, it's all good. IIRC this was part of the wish behind block-pointer-rewrite. I see this as really being a problem for commercial entities which have to preserve snapshots for legal reasons as their old snaps are written with the old compressor (hospitals, law firms, financial services companies, gov, etc). Also 100% w/ you that this belongs inside ZFS itself as an automated function, not an external script which interacts with the VFS to perform it (in another thread i had suggested reprocessing all writeable blocks if a new compressor version is detected using a background thread). That said, thanks for paving the way with manual approaches for this as the historical reference of that working will probably help in getting some version of it into ZFS itself.
@DemiMarie - i think that the community's faith in what has materialized as the OpenZFS org is what's in-question here. There are people talking about corporate interests and secret clubs. I dont have a dog in that argument, but its not the first time such accusations have been leveled on this issue tracker so is indicative of a lack of trust in the leadership. Whatever motivates the governing authority to move forward with code or to commission the writing of new code seems to be a bit out of sync with the big ticket (and not @ all sexy) items around low-level design and functionality/performance that require a ton of work, the majority of which will never be seen by anyone, just experienced by all users. I deal with this in Metasploit all the time - people are retiscent to make big changes to /lib, even got heated on a few occasions over the last decade (but we've all evolved and learned to work across the corporate/community divide). The idea of having an external entity fund and guide development separate from the interests already entrenched in the OpenZFS ecosystem strikes me as an end-run around the conjecture and acrimony possible here, directly toward productive ends.
@rincebrain - the main argument i had for making compressors take versions into account is security-oriented. Concern being that without formal mathematical proofs of safety in data manipulating routines (esp those which malloc), hard-coding them into a very long-lived stack creates a potential for architecture-breaking exploits (imagine someone figures out an inverse zip/zstd/lz4/etc "bomb"). Now that leadership has decided that its ok to replace the compressors, that concern is mitigated and replaced with the "we have data which we cannot rewrite for dedup" thing; and IMO (being a security person), that is a much smaller problem (as @BrainSlayer points out, its not even universally applicable so long as you can perform writes on the blocks in-question). If we ever get something that could re-compress RO snaps, then the compressor versioning thing might be viable as a discussion point again but at that point you could just check if your compressor outputs the same thing as the on-disk block (presuming all the on-disk sums are good) and recompress if it doesn't match (with a subsequent decompress to verify). Expensive, but solves the storage ballooning issue.
@jittygitty - how do you get everyone putting money into the pot to agree on the specification and how it is spent? Take the async-DMU work: there are 10s of hours of just code review by a high-end C dev to take up before a specification can be properly written. Who pays for the amorphous deliverable of a code review on a moving target (master branch) and old PR code - it should be done by someone who knows the code, but if we're hiring from outside, then it's that much more time/cost for which no functional deliverable can materialize until a later phase. Do you pay them for review + spec creation expecting another contractor to then take up and execute that spec, and how do you pre-qualify what a proper specification must include? Do you pay them when the whole effort is done and approved for merge (does their pay depend on leadership caring about the PR)? Or do you write detailed milestones based on the specification provided by phase 1 and then track their commits and work notes in Redmine or something (GH took the "KISS" approach to work package tracking, not my favorite for complex efforts with dependencies)? If you put money out for people to just take, they will just take your money - short of strict deliverable requirements against which they are to be paid, its a miasma of con and grift out there (trust and money/power don't mix). Lastly, qualified C devs are in short supply; so we need to put up a lot of money to rival actual contract offers which these folks (teams probably for the bigger stuff) are getting in their day-to-day work and a professional working environment in which they can operate to provide the quality of pay+ecosystem in which they are willing to operate. A lot of money up-front means that a neutral, and legally bound, third party needs to administer it to avoid one of the organizers taking it and tanking the project. This would have been easier back in the wild-west FOSS days, i admit, but i think we need to operate in today's reality where immediacy of gaining the upper hand and short-term gains rule the general mentality.
BrainSlayer
commented
2 years ago @sempervictus older snapshots will still work. i mean rewrite of older blocks is a option and not forced and what counts is the decompressed content, not the compressed. just my thoughts. of course we can also include versioned versions of zstd for each new version. but i think this will result in a very big driver in 10 years. its all a question of philosophy.
jittygitty
commented
2 years ago @rincebrain In terms of 'architectural complications' I think maybe you meant to say BPR (Block Pointer Rewrite) instead of BRT (Block Reference Table)? The elusive BPR which is extremely difficult was I think meant as the panacea miracle fragmentation fix.
@sempervictus Sorry but you misunderstood the "pooled pot" so to speak. The money would NOT be pooled "literally" at all. Each of us would fund PRECISELY the issues/features we want in the way we want it, each of us controls the requirements for our own funds to be released. Also, besides making a profit, another reason these platforms have arbitration so that if coder insists that they complied with "all your requirements" but you're not releasing the funds from escrow, the arbitration steps in and decides after each presents their case. I have hired coders this way for about two decades and only once or twice ended up in arbitration, which I easily won. On these sites even in not so cut and dry cases, the 'employer/funder' usually gets preference. The only difference with BOUNTY sites, which I have not used, is that its like a contest and whoever accomplishes what you asked for "first", he is the one that will then be awarded the money that YOU put in escrow. Now if OTHER people put money in escrow and chose to use YOUR requirements for same feature then they can get their money too, but they can just as well have "further" stipulations to meet to collect on their funds also. So as you can see it is very "flexible". The "pooling" of funds I was talking about is simply that if we tag/link ourselves as part of same team/project on these sites, the sites will ADVERTISE next to our "TEAM NAME", the TOTAL amount available from all of us team 'members', but each team member has "control" over what requirements must be met by Coder/Bounty-Hunter for their specific funds to be released to that coder. So its quite safe in that sense that if the coder fails to implement your requirements as you specified, you do NOT have to release the funds to them. In reality it is the Coder/Bounty-Hunter who has a whole lot more risk and must have some trust, especially on "copyleft" work etc. Of course it would not be fair to a coder that has met all your requirements for functionality, to then say, oh wait the review guys don't feel like taking new PRs now so you can't get your money. You could of course put this in your requirements, but who would then take your Bounty Seriously? You can take their working/code solution and then PAY/Donate/Fund the Review guys to actually Review the PR. Or you can offer the coder EXTRA MONEY if/when his Pull-Request is accepted by maintainers.
Sadly, from experience, I will have to agree with you that kernel and file-system level C coders are in short supply, but there are some out there, ready for work, that I know from experience also. And because these Bounty freelancing sites have coders from all geographies, some are much more affordable than citizens of the USA or Germany or other affluent nations. Again, I have hired many over the years and haven't had any of the issues you mentioned that you were concerned about, but I'm open to hear anyone's "bad story" as well to learn from. Anyway I hope that I clarified why everyone's funds would be "safe" and under their full control as to what met requirements trigger release of "their" own money. Any other objections, or any willing to try?
sempervictus
commented
2 years ago @sempervictus older snapshots will still work. i mean rewrite of older blocks is a option and not forced and what counts is the decompressed content, not the compressed. just my thoughts. of course we can also include versioned versions of zstd for each new version. but i think this will result in a very big driver in 10 years. its all a question of philosophy.
Agreed, at the rate of ZSTD's progress that could get gory real quick. This is where the idea of keeping uncompressed sums would be handy since any prior (or i guess other) compressor version which was used to compress the data would result in the same decompressed sum but since the compressed sum would be different it would merit a rewrite of the block. I think that's where the snapshot paradigm is a bit problematic in that the immutable state should really apply to the logical data contained in the snap (since the snap can already be sent/recvd), to permit re-compression. It could probably be done as a scrub-like function in the background but since the output sizes would vary i'm not sure how deep the rabbit hole goes on DMU-level approaches.
@jittygitty - thanks for the clarification. Agree that 'race to the finish' doesnt seem like a good paradigm here - works great in offsec-land, not so much here. In terms of where developers originate - there are definitely places where cost of living is lower and tech industry is thriving, so we should be able to use globalization to our advantage in that regard. Language barriers are "a thing" though in some cases which can make nuances of specification difficult to resolve. Hopefully we've enough folks in the community to act as translators when needed :smile:.
jittygitty
commented
2 years ago @sempervictus Yes you are right about language sometimes, since I've experienced a few times where I had a great coder but it was taking more of my time to explain due to language. Recently though this hasn't been as much of a problem, maybe with social media and English being so rampant that with the increased internet access perhaps people are learning English faster. Basically the talented and affordable programmers I've found are from Asia (all of it, several nations) and also Eastern Europe.
zfsbot
commented
2 years ago a majority of the capable developers who can, have, or are already working on OpenZFS, are just too busy with work they're already contractually obligated to perform.
you might not have been here for very long. we've lost more developers than we've gained in 10 years. it's not a question of funding, there's always been plenty to go around. lots of companies who can and do finance ZFS work. but the short term pooled projects you're proposing had better be extremely lucrative to draw anyone over from their reliable salaries.
DemiMarie
commented
2 years ago you might not have been here for very long. we've lost more developers than we've gained in 10 years. it's not a question of funding, there's always been plenty to go around. lots of companies who can and do finance ZFS work. but the short term pooled projects you're proposing had better be extremely lucrative to draw anyone over from their reliable salaries.
Are any of those companies willing to hire someone full-time?
jittygitty
commented
2 years ago @zfsbot I think that I, (and maybe @DemiMarie also?), question the plenty of funding to go around. Maybe we should ask these companies then to put up some good money in funding some issues or features via such Bounty system, its better bang for the buck if it works out and if it doesn't they don't lose their money. Are you saying they posted jobs but found no match?
As I mentioned, although you're correct that maybe we can't afford a seasoned C/Kernel/etc expert who's a US citizen and wants a $250k-$500k salary with benefits, you can find someone from overseas who's happy to work for an affordable wage.
Describe the feature would like to see added to OpenZFS
How can we VOTE or do a POLL on here on which Crowd-Funding BOUNTY rewards platform we should use on here to fund issues?
This was originally from my Issue at #13349
"Can we do a POLL here to see what BOUNTY platform we should use, where all interested in reflink/offline-dedupe can fund?
I was looking at BountySource https://bountysource.com/ but heard lately that after some time you can't get your unclaimed funds back from them. I just found out about https://www.openbugbounty.org/ Would be good to have a poll and decide on one since there's so many: ( https://gitpay.me/ , https://gitcoin.co/ , https://bounty0x.io/bnty , https://issuehunt.io/ , https://tip4commit.com/ , https://liberapay.com/ , https://flattr.com/ , http://en.goteo.org/ , https://gitcoin.co/ )
Would be great to get those would-be ZFS patron funders together to help fast-track this issue and others. So if you're able and willing to contribute funds or if you're a bounty hunter, would be great to get feedback and preferred "platform" to use. "
How will this feature improve OpenZFS?
There are a LOT of issues and feature requests, some would be a huge boost to ZFS for many. Having a BOUNTY rewards platform where we can all agree on where to fund issues would I think attract more talent to start working on ZFS, we can definitely use more code contributors, can't we? What better way to gain new code contributors than offering some rewards/bounties for various work?
Many of us, myself included, have for a long time hoped that Github would add the feature itself built into the system. Unfortunately this has not happened, and each of us using separate platforms will spread out the Funds/Rewards too much and weaken the incentive. Although the argument can be made that using a couple of the top platforms might be best to increase exposure and get the attention of the most gifted programmers, especially if when a reward is claimed on one platform we can remove it from others.
So the question is, should we agree on ONE platform? Or which of the top/best platforms? Which one/s?
I've listed some of the ones above that I've become aware of, are there even better ones I've missed?
These should be BOUNTY/rewards based systems, though crowd-funding "DONATION"-based platforms (such as Kickstarter, indeigogo, gofundme, Patreon, etc) would be good also and the funds could be used from those to fund the BOUNTY based systems.
I hope the leaders here are ok with this, so also look forward to hear their feedback also: @ahren @wca @behlendorf @pjd @scsiguy @allanjude @delphij @lundman @mmatuska @grwilson @happyaron @ryao Please feel free to notify others I may have missed! (I can't find Chris Siden handle, all taken from: https://openzfs.org/wiki/Contributors )
There's no immediate need to have an official OpenZFS account to fund from, though that wouldn't be a bad idea. Until then though we can simply agree on a platform or platforms to use and anyone can fund there and tag "openzfs" but having one account allows pooling the funds which can be beneficial. But I think some platforms allow funds pooling (ie for the sake of bounty hunters looking for well-funded teams) by allowing us to links as being part of the "same team" ie openzfs-team etc.
Would be exciting to get this going since there are many issues on here that I think a lot of us would be excited to start FUNDING! : )
Additional context
If there's no way to poll I guess someone can propose one in a comment and everyone can vote by Thumbs-up on the comment?
Update: Donation sites would need OpenZFS leaders to setup one "official" account with them, or for individual "code contributors" to have their own individual accounts.
This poll is mainly for Bounty/Rewards type platforms where any of us can post our own Rewards/Bounties for a "Specific Issue" and Bounty-Hunters/Coders can compete and claim reward/bounty if they Solve Issue or Implement the Feature Request.