Repo & Maintainer Status & Availability

[tannerlinsley]

Hello maintainers!

I hope you're all doing well. Many of us feel that we haven't heard from you in very long (aside from the few issue comments here and there), and are curious concerning the future of crossfilter. We've heard that Square is no longer using it internally, or that it has become less of a priority to them, and that some of the maintainers have moved on to other ventures. Can anyone give us any details in this area? There are some great PR's and people that want to contribute to crossfilter and (despite it's continuous growth), I worry that it gets the attention it deserves.

Cheers!

[RandomEtc]

Fair question, thanks for asking!

True: nobody at Square is actively maintaining this library right now, and we don't use it in the Square Dashboard where it was originally deployed (we moved all computation server side again).

That doesn't mean nobody is using it. I'll ask around and see if someone is willing to help triage issues and merge down good PRs. If not, we'll look for a different solution (perhaps a couple more external maintainers to help Jason out!)

[gordonwoodhull]

Does crossfilter have any bugs, besides people shooting themselves in the foot and getting stack overflows with NaNs? I think in a sense it is complete.

I can't contribute due to the copyright assignment agreement. That's something that creates a sea of red tape in corporations.

[tannerlinsley]

Not that it has bugs, or that it is incomplete. But there are some features that have yet to be created for it. Simply we just want to make sure it's getting the attention it deserves. Also I volunteer myself and @esjewett as contributors. On Wed, Apr 29, 2015 at 4:47 PM Gordon Woodhull notifications@github.com wrote:

Does crossfilter have any bugs, besides people shooting themselves in the foot and getting stack overflows with NaNs? I think in a sense it is complete.

I can't contribute due to the copyright assignment agreement https://github.com/square/crossfilter/blob/master/CONTRIBUTING.md. That's something that creates a sea of red tape in corporations.

— Reply to this email directly or view it on GitHub https://github.com/square/crossfilter/issues/151#issuecomment-97608341.

[tannerlinsley]

At least to triage issues and give the repo a sense of activity. On Wed, Apr 29, 2015 at 5:13 PM Tanner Linsley tannerlinsley@gmail.com wrote:

Not that it has bugs, or that it is incomplete. But there are some features that have yet to be created for it. Simply we just want to make sure it's getting the attention it deserves. Also I volunteer myself and @esjewett as contributors. On Wed, Apr 29, 2015 at 4:47 PM Gordon Woodhull notifications@github.com wrote:

Does crossfilter have any bugs, besides people shooting themselves in the foot and getting stack overflows with NaNs? I think in a sense it is complete.

I can't contribute due to the copyright assignment agreement https://github.com/square/crossfilter/blob/master/CONTRIBUTING.md. That's something that creates a sea of red tape in corporations.

— Reply to this email directly or view it on GitHub https://github.com/square/crossfilter/issues/151#issuecomment-97608341.

[derekperkins]

I'd love to see some more activity. There are a few PRs that have been submitted that I'm very interested in, like adding support for more dimensions #75 and one that I can't find by @esjewett that allowed you to remove rows that don't match the current filter.

@esjewett also has https://github.com/esjewett/reductio that is a good place for a number of things that might be useful but deemed out of the purview of core crossfilter. Maybe moving crossfilter to an official organization so that there could be additional plugins brought into the official crossfilter ecosystem.

[tannerlinsley]

I second that On Wed, Apr 29, 2015 at 5:28 PM Derek Perkins notifications@github.com wrote:

I'd love to see some more activity. There are a few PRs that have been submitted that I'm very interested in, like adding support for more dimensions #75 https://github.com/square/crossfilter/pull/75 and one that I can't find by @esjewett https://github.com/esjewett that allowed you to remove rows that don't match the current filter.

@esjewett https://github.com/esjewett also has https://github.com/esjewett/reductio that is a good place for a number of things that might be useful but deemed out of the purview of core crossfilter. Maybe moving crossfilter to an official organization so that there could be additional plugins brought into the official crossfilter ecosystem.

— Reply to this email directly or view it on GitHub https://github.com/square/crossfilter/issues/151#issuecomment-97613401.

[jefffriesen]

I wonder if some people have moved onto other solutions for some reason. Lodash keeps getting faster and more functions. There are now lazy evaluated libraries (lazy.js and immutable.js) which are really performant for large data sets (I don't know if lazy evaluation would make a difference here). transducer.js is another approach that could be fast.

Is there less activity in this library because it's pretty solid and mostly feature complete? Or is it because people are starting to use other options? Does anyone have a sense of that?

(btw, It would be awesome to see some benchmarks between crossfilter and the equivalent operations in lazy-eval libs and lodash for small, medium and large datasets)

[gordonwoodhull]

Maybe moving crossfilter to an official organization so that there could be additional plugins brought into the official crossfilter ecosystem.

An organization would be sweet, with copyright staying with the contributors, as we have done with dc.js. The only reason for copyright assignment is to make it easier to change the copyright later, and that is unlikely to be needed with crossfilter.

[gordonwoodhull]

@jefffriesen, has anyone built up the same cross filtering functionality on top of lodash? That would not be trivial but would indeed be interesting.

[derekperkins]

@jefffriesen - I feel like dc.js and other libraries are making crossfilter more accessible and hence more popular.

The core functionality is awesome, so there aren't major core changes that need to be implemented. Mostly just some tweaks and a few feature additions.

[jefffriesen]

@gordonwoodhull I don't know if anyone has built up something like a .crossfilter() or some combinations of .reduce() and other things. I would be interested to know too.

@derekperkins I agree with you that dc.js has made crossfilter more popular - that was when I first used it.

I could imagine someone taking a step further in functionality like reductio. Or I could imagine performance gains for very large datasets or many dimensions, by offloading the computations in a web worker or doing lazy evaluation. But I don't know what's possible or what's out there.

I am very fond of this library though...

[tannerlinsley]

I love crossfilter and reductio and even the idea of having a crossfilter ecosystem for plugins or helpers like it. I'm already in development of a dimension and group management micro library that will integrate with both. It would be great to have a home for it.

Who is technically at the reigns right now? On Wed, Apr 29, 2015 at 9:07 PM Jeff Friesen notifications@github.com wrote:

@gordonwoodhull https://github.com/gordonwoodhull I don't know if anyone has built up something like a .crossfilter() or some combinations of .reduce() and other things. I would be interested to know too.

@derekperkins https://github.com/derekperkins I agree with you that dc.js has made crossfilter more popular - that was when I first used it.

I could imagine someone taking a step further in functionality like reductio. Or I could imagine performance gains for very large datasets or many dimensions, by offloading the computations in a web worker or doing lazy evaluation. But I don't know what's possible or what's out there.

I am very fond of this library though...

— Reply to this email directly or view it on GitHub https://github.com/square/crossfilter/issues/151#issuecomment-97646711.

[RandomEtc]

I'm the Square maintainer for crossfilter, and @jasondavies is also a maintainer who independently merges PRs and new releases. In 99% of cases over the last 2+ years I have just deferred to Jason.

Still asking around internally at Square for volunteers (not opposed to more outside maintainers but it's more work to manage, so starting locally). Give us a day or two to figure it out.

@gordonwoodhull I will also ask about the agreement in case there's room for an update there. Is any agreement an impediment, or something about our specific one? Understand either way, just curious if it can be fixed.

[derekperkins]

@RandomEtc Thanks for being so responsive.

FYI, crossfilter was available as an organization name, which I just registered on a whim the other day, when @tannerlinsley and @esjewett and I were talking about some extensions we'd like to make. It's available if there is interest in moving it outside of square.

[gordonwoodhull]

@RandomEtc I would say in many organizations it is only moderately difficult to get internal approval for contributing to a project under some well known open source license.

But it is near impossible to get approval to sign some other agreement that lawyers would have to read.

(Spoken from my limited understanding of IP and organizations.)

[esjewett]

Hi @RandomEtc and all. Thanks for discussing this. I'd be happy to help maintain if that would be helpful.

I'm definitely on-board with the general philosophy of keeping Crossfilter as lean as possible and keeping complexity in helper libraries, but there are some things that it would be extremely helpful to address internally that have not been getting attention. I'm thinking here of issues like #109, #92, and #75. It would be great to have a small team of maintainers and major users of Crossfilter in place who could work together to review, improve, and merge these sorts of changes.

As far as I've seen, there's nothing else out there like Crossfilter. I'd like to see it thrive as I think it's a very important piece of any type of interactive visualization of more than a small amount of data.

[RandomEtc]

On #109 it seems like there was some support from Jason already, but no code. I would create a branch and try it, and get buy-in from other contributors. Forgiveness not permission. It would mean cutting a major release number if it breaks the API, but that shouldn't be an issue if the pay-off is worth it.

92 certainly looks like cause for concern since I've ignored it for 18 months. Apologies for that. Folks who are following along, it would be helpful to review and test this PR if it's a strong candidate for merging. I have no opinion on it except what I already shared :)

75 needs reviewing, verifying and benchmarking again following @sciyoshi's fixes. Please help.

[esjewett]

109 has code (see the linked branch), but since Jason wanted to take another approach I was trying to get buy-in before creating a pull request. I don't want to waste effort on this. It doesn't break the API, just extends it. If you (@RandomEtc) think you'd be OK with merging that code, I'll submit a pull request ASAP.

On the other two, it's a bit of a horse/cart problem. If there is not active maintenance of the project, it is difficult to justify putting significant effort into review that may never be looked at. I look forward to hearing what Square decides on that front. Sorry to be difficult on that, but I think that's the key issue to be resolved.

[RandomEtc]

I freely admit there is no active maintenance from Square on the project at this time. I'm able to drop-in to discussions like this quickly, and can provide stylistic and process guidance, but I can't commit to properly reviewing PRs if that's what it takes to unblock them.

Still asking around internally, that will give us the info we need to decide what happens next. I'm open to "blessing" a fork as well, which is what we did with Cube when it became clear we weren't going to actively maintain that.

[esjewett]

@RandomEtc No worries, we all know how it is with finding the time for these things! And I didn't mean to cast blame as far as ongoing maintenance, just to make clear that the issues of getting proper review and involvement from the maintainers is related in my mind.

I'm wasn't familiar with what happened with Cube. Good to see there is some precedence for dealing with these types of projects.

[wssbck]

@RandomEtc Out of curiosity - can you share some details as to why you moved dashboard aggregations to the backend? The added interactivity of frontend calculations was not worth it after all?

[RandomEtc]

Many reasons. Some including: data volume, mobile performance, parity between results on different reporting platforms (web, app, email, etc).

[esjewett]

Hi @RandomEtc, just checking in to see if there is any update from the internal discussion @ Square? Cheers, Ethan

[RandomEtc]

No volunteers inside of Square. I'm not sure of the best steps forward, I'll find a time to swap notes with @jasondavies and see if we can identify one or two people to assist.

[RandomEtc]

Aside: it sounds like a lot of contributors are finding crossfilter via dc.js? We've also had several issues filed here that turned out to be dc.js issues. Would the dc.js maintainers consider forking crossfilter for their purposes and accepting patches directly there?

[esjewett]

Thanks for the update. I think @gordonwoodhull would be the key person to ask regarding the dc.js angle. I'd be up for helping maintain a dc.js organization fork or another fork - whatever is agreed.

[derekperkins]

@RandomEtc - I don't love the idea of forking crossfilter into dc.js or into that organization. Not that dc.js isn't great, but there are still plenty of people using crossfilter outside of that context, myself included. I'd love to have @gordonwoodhull and the dc.js team heavily involved, but I think crossfilter is big enough to warrant separate maintenance.

My vote would be to officially transfer this repo to a crossfilter organization. If square doesn't want to do that or there are other issues preventing that, then next best would be a blessed fork to that same organization.

Whatever happens, I think a key thing would be to change the licensing to something more open so that we don't have any other contribution issues.

[gordonwoodhull]

Thanks for the update, @RandomEtc. Yes, dc-js could host crossfilter, but it might be more appropriate to put it in its own organization as @derekperkins suggested.

Since the dependency is really the other way and lots of people use crossfilter without dc.js.

Either way I'll be glad to help maintain it, as long as there are few other maintainers and no contributor agreement.

[derekperkins]

@gordonwoodhull - Great minds think alike :)

[derekperkins]

My vote for new collaborators would be @esjewett, @gordonwoodhull & @tannerlinsley.

[tannerlinsley]

I'm on board.

On Tue, May 12, 2015 at 12:00 PM Derek Perkins notifications@github.com wrote:

@gordonwoodhull https://github.com/gordonwoodhull - Great minds think alike :)

— Reply to this email directly or view it on GitHub https://github.com/square/crossfilter/issues/151#issuecomment-101366448.

[sylvinus]

I'm just a user at this point, but very glad to see this happening!

[derekperkins]

@jasondavies : I'm very curious to hear your opinion on the matter, since you're the authority at this point. Are you interested in continuing to be the lead maintainer of the project? Do you have plans for future crossfilter development?

While we are all interested in helping crossfilter grow, I think all of us would defer to your judgment at this point. I don't believe that any of us are looking to "pollute" the repo with a ton of feature requests. I think the vision is more about building the ecosystem around the core:

• https://github.com/esjewett/reductio
• https://github.com/dc-js/dc.js
• @tannerlinsley has a library similar to dc.js that interacts with https://github.com/nnnick/Chart.js

In terms of core crossfilter changes, I think there are only a few outstanding issues that we'd like to see resolved.

All that being said, I really believe that the best solution is to transfer the existing repo to a crossfilter organization to legitimize some of these plugins and to aid in the discoverability. This should also allow for us to continue to support existing npm and other systems pointing at the existing url.

@RandomEtc : Is there any resistance internally to pushing the repo out to an org?

[gsklee]

+1, hope to see this great project moving forward again.

[RandomEtc]

Sorry for the delayed update here.

I spoke with Jason and we're in agreement at a high level: as @gordonwoodhull speculated above, we consider crossfilter essentially "finished" at this point (being targeted at the use-case on the demo page: linked charts of dimensions from a simple flat dataset). I share this as plainly as possible so you know why Jason and I are not picking up the pace on PRs on this repo.

We completely agree that there are possible extensions and API improvements that would be worth putting into a "crossfilter" v2. But we also agree that if we were doing that we would probably write something from scratch using the lessons learned from v1. Neither of us is planning to start such a project.

I am investigating if any of Square's other projects have ever spun out into a github org. I doubt that it will happen - it's difficult because of our requirement for contributor agreements. It's more likely we would encourage contributors to organize among themselves (something like "crossfilter-contrib" would make it clear it's not Square's thing, and could always be renamed in future). If the participants on this thread want to go ahead and do that, it's probably the best way to focus the energies and enthusiasms of the group.

[derekperkins]

Thanks for the consideration and the update @RandomEtc - we really appreciate everything that you've done. I don't think that anyone has plans at the moment to do a full v2 rewrite. The updates are more related to building out the ecosystem of plugins / extensions, and leaving the core API mostly unchanged.

I definitely understand the legal difficulties associated with actually spinning it out. We went ahead and forked into the crossfilter organization https://github.com/crossfilter/crossfilter. We will make it very clear that the original code was written at Square, but that this is not an official project supported or maintained by Square. Once we get things set up on that end, do you have any concerns "blessing" that fork, similar to what you did with Cube?

[RandomEtc]

Just closing the loop here...

Self-organizing an org makes sense. That's what I would have done. I would personally prefer it be called something else (add "-contrib" or something) but I'm not speaking for Square here. Speaking for Square - please unpublish http://crossfilter.github.io/crossfilter/ and/or remove the Square logo from it and make it clearer that it's a fork. It's probably best to remove the gh-pages branch from the fork until you're ready to cut a new release. Likewise please update the README to not be written as Square. Thanks!

Before linking from Square's crossfilter repo I'd like to ensure that meaningful additions/features are happening - the forks of Cube were significantly modified/enhanced before we linked to them. Keep me posted once the initial merges settle, I'll make it happen!

Thanks for understanding and keeping the energy going!

[esjewett]

@RandomEtc Thanks for closing the loop and the feedback. We're still getting things together and moving a bit slowly, but I've gone through and tried to update all the documents in the main repo and gh-pages to remove Square branding and "point-of-view" except for the copyright notices. I've also added a sentence in the first paragraph of the gh-pages site to indicate that it is a community fork and link back to the original project. Sorry for the slip there.

Regarding the renaming, that is a bit difficult. In some ways, a new name would be ideal (for many reasons including reduced confusion, Bower/NPM registration, etc), but this is also meant to be a continuation of the same project. crossfilter-contrib, at least in my mind means additional components that work with the original Crossfilter library, and that's not what this is, or at least not only. So, I think it may be a slow process to rename. We'll talk about it and figure something out!

Thanks again for your understanding and guidance with this.

[dderiso]

@esjewett @RandomEtc Thanks for all the hard work! Is it correct to summarize this as Crossfilter's home for active community development is now: https://github.com/crossfilter/crossfilter ? If so, it might be good to put a note on the readme of this repo so devs know where to submit pull requests. Thanks again for everything, love this library.

[esjewett]

@dderiso I'd say that https://github.com/crossfilter/crossfilter is open to pull requests and issues. Probably best to first open an issue to discuss as we only want to make changes that are really required in the core library. We are working very slowly though. It's probably quite premature to say that it is the home for community development as that's the kind of thing I'd prefer to show rather than say, and I don't think we've shown that yet.

[tannerlinsley]

Also the great thing here is the crossfilter organization. Sure we don't know if this is the home for anything yet, but at least there is room here for not only new features but also plugins, and enhancements that fall under the crossfilter umbrella. On Thu, Aug 6, 2015 at 6:37 PM Ethan Jewett notifications@github.com wrote:

@dderiso https://github.com/dderiso I'd say that https://github.com/crossfilter/crossfilter is open to pull requests and issues. Probably best to first open an issue to discuss as we only want to make changes that are really required in the core library. We are working very slowly though. It's probably quite premature to say that it is the home for community development as that's the kind of thing I'd prefer to show rather than say, and I don't think we've shown that yet.

— Reply to this email directly or view it on GitHub https://github.com/square/crossfilter/issues/151#issuecomment-128548864.

[derekperkins]

@RandomEtc - We're reporting back after a lot of work has gone into the https://github.com/crossfilter organization.

• crossfilter: @tannerlinsley & @esjewett added support for nested arrays https://github.com/square/crossfilter/issues/5
• reductio: @esjewett & @nordfjord added grouped aggregations
• universe: @tannerlinsley created a mongo-like query language + automatic dimension management
• async: @esjewett has crossfilter working in a web worker #116

@gordonwoodhull & @r4j4h have also been very helpful along the way.

We're all very committed to continuing to push the crossfilter ecosystem forwards. With @gordonwoodhull & dc.js onboard, a significant portion of active users will likely be seamlessly and silently migrated, and we'd love to be able to make the transition for them and all other crossfilter users as simple as possible. Here's what I think would be the best course of action moving forwards, and I'm more than open to other suggestions.

1. If possible, transfer this repo to the crossfilter org (lawyers permitting)
2. If not, post a deprecation notice similar to cube and point people at crossfilter/crossfilter
3. Redirect npm & bower to the org version of crossfilter - We guarantee BC for 1.x.

[gsklee]

Glad to hear this, kudos to everyone involved for moving this project forward!

[RandomEtc]

@derekperkins @tannerlinsley @gordonwoodhull et al - thank you for keeping the dream alive. I've chosen option 2 as the easiest for now, and added a note to the README on the square version at https://github.com/square/crossfilter/blob/master/README.md

I have asked internally about unlocking/transferring the crossfilter npm and bower packages so that newer versions can be published there, and about transferring the repo itself to allow github to redirect everyone. But I make no promises about following through with this - please act as if it won't happen, I'll ping you if the situation changes :)

Hopefully it is clear that crossfilter/crossfilter is the home of active improvement. Thanks to all who weighed in on this thread. Please take further discussion to an issue there! :chart_with_upwards_trend: :raised_hands:

[derekperkins]

@RandomEtc Thanks for following up! We'll keep our ears up in case something changes on the legal front, but we definitely understand the power of inertia at a big company. :)

Our thanks go out to you, @mbostock, @jasondavies and everyone else involved with getting crossfilter to this point!

[RandomEtc]

Don't thank me for sitting on it for 3 years followed by a flurry of copy-paste issue closing :grimacing:

[gordonwoodhull]

Thanks @RandomEtc!

There is a wealth of information in the issues on this repo, so let's link back here when we continue these conversations. Looking forward to continuing the amazing work done here!