Open davidanthoff opened 7 years ago
@nalimilan I also have a a branch on CategoricalArrays that uses DataValue. I almost feel the best way to handle that would be to just move that stuff into DataValues.jl for the time being. The only other option would be to create another package DataValueCateogoricalArrays, but given that all of this might go away after julia 1.0, that seems not so good? Not sure, though.
Yes, that could probably live in DataValues.jl, though it's obviously up to you. As for the future of DataTables, I think we'll have to discuss that with all involved parties.
As for the future of DataTables, I think we'll have to discuss that with all involved parties.
Yep, totally agreed. Who should weigh in on that? Maybe we can ping them on this issue?
I'd say anybody interested enough in that area to have subscribed to the #data Slack channel?
Bump.
Ideally (from my point of view) we would move this repo over to https://github.com/queryverse and I would take over maintenance of it.
Could we just move ahead with that, given that no one objected for almost a year? I believe I did mention this back then on the data slack channel. Or should I mention it there again and then we'll wait a week to see whether anyone has an opinion about it? I almost suspect that no one cares anymore, given the current state of this package ;)
Here are a few of my thoughts/questions/concerns:
It just makes me very uncomfortable that this package would be repurposed into something that would confuse users and probably not be able to keep up in terms of latest and greatest in DataFrames. I think a much more tenable approach would be to take the simple type definitions and basic indexing methods and put them in DataValues.jl to help reinforce the idea that it's an extension of DataValues and it's approach to missing values as opposed to a full-fledged DataFrames-like package.
I'd like to revisit this. We plan to make a DataTable
type a central piece for the Queryverse story going forward. We will use the name DataTable
for the type, so the question now really is only in what package that type will exist.
I don't want to put it into DataValues.jl, for several reasons: 1) the DataValue
aspect is not going to be the central point for this going forward. Yes, it will be based on DataValue
, but that is only a small part of the plan. 2) There are other table types like IndexedTables.jl and JuliaDB.jl that use DataValue
, and it seems quite wrong to sneak a full table type into their dependency graph. The more general point here is that a large number of packages depends on DataValues.jl, and most of them have absolutely no need for a table type, and so I want to keep a package that is low in the stack (like DataValues.jl) lean and not overload it with stuff that is only useful for some users.
In terms of user confusion, I don't see how putting a table type named DataTable
into a package Someothername.jl
is better than just putting it into DataTables.jl
. The general pattern throughout the ecosystem seems to be that a) each table type has its own package, and b) the package name is the plural of the type name of the table type. I'm suggesting that we follow the same pattern here.
Re @quinnj's concerns about maintenance and feature scope: we are three core devs on Queryverse here in Berkeley now, and I expect that this will ramp up significantly going forward. I would also suggest that the track record of Queryverse be taken into account here: it has been around for a long time, it is expansive, and I think it is fair to say that it is pretty well maintained. We plan to handle the DataTable
type in the same way, and so I expect it to be as well maintained as the rest of Queryverse.jl.
I think at the core, though, the question here is really: what is the value of keeping a good name like DataTables.jl dormant? Why not reuse it for something useful?
Why can't you just use DataFrames?
We want to do a fair bit of DataValue
specific stuff, and my understanding is that DataFrames.jl doesn't want to take that dependency, nor adopt that approach.
That used to be the case a long time ago, but DataFrames can now hold arbitrary columns. So it could be Vector{DataValue}
or whatever.
We want to do things beyond just storing DataValueVector
in a DataFrame
. The things we plan to do require DataValue specific code in lots of places where I think it wouldn't make any sense to have it for a table that is primarily Missing
based, like DataFrame
.
But the DataValue
aspect is only one of many things we want to do differently. I think in such a case it is better to just start a new package, rather than me storming into DataFrames.jl and trying to change lots of things in a pretty fundamental way. In particular because I think that would be really harmful to DataFrames.jl, it is a great package and just because we want to do things differently in queryverseland does not imply that it would be a good idea to even consider them for DataFrames.jl as well.
I think at this point it probably is not super helpful to discuss whether Queryverse should have a table type or not. We thought about, planned it out pretty conclusively and from my point of view we are committed to having one. I think it probably makes more sense to keep this discussion on the question whether the DataTables.jl package name can be recycled or not.
I'd be interested in hearing what you would like to do differently and why. Let's keep that for another discussion, but I think it's worth having it as we already have too many table types given our limited workforce IMHO (I think developers of other packages would also agree).
But anyway. I'd really not like to see a new package reusing the DataTables name, based on yet another approach from what DataTables experimented at the time it was forked. It's been painful enough to fight the confusion due to the coexistence of DataFrames and DataTables (we still hear complaints about that on forums). Having to explain that there was an "old DataTables" and now a new one, with different concepts and maintained by different people doesn't sound great for clarity. Couldn't you find a more explicit name which reflects the strengths of the new package (like TypedTables or IndexedTables do)?
I agree that it's a shame that after planning/studying things out, you've chosen to not collaborate w/ the rest of the data ecosystem and do something different entirely. I've voiced it elsewhere and I'll voice it again that it's the users who lose the most when multiple, almost identical packages exist that essentially provide the same functionality/purpose. I understand there are fundamental challenges in open-source w/ communication, collaboration, coordination, etc. but I've always loved the Julia community spirit of working together to improve things and work towards common goals. Obviously, there's always room for alternative approaches and experimentation.
In this case specifically, I agree w/ @nalimilan that the potential for confusion will be great among users. Why not create a new package named DataValueTables.jl? That seems ideally precise in that it's a table-like package based on DataValues.
I wrote this:
@nalimilan wrote this: