Gmousse
commented
7 years ago Hi @dragoljub,
Your question is really interesting. As you said, the DataFrame object is not really column-based data structure but it's actually row-based.
To write this codebase I had two options:
- Row-based: Define a DataFrame as a collection of rows having a schema. It's used by Spark (or other functional, immutable, big data structures). The API should be functional, based on maps and reductions.
- Column-based: Define a DataFrame as a collection of columns having a type. It's used by R, pandas (in python). The API should be based on vectorial computations and usually mutable.
I have chosen Row-based because:
- I think the functional paradigm (and the Spark implementation and API) is more in accordance with the Javascript evolution (and future?), and also with the way I work.
- The row is easier to maintain the dataset structure. Indeed, we don't have to work with index in Column vectors to manipulate row data (which can be a pain in a group by for example).
- It's better (in terms of computation and usage) to make reductions taking multiple columns in account to filter rows, to sort them or to modify them.
- It's easier to make rows immutable.
- (as you said) It's perfect in front-end applications (vizualizations, list of data, filtering...).
- Rows is the best format for parallelism (It will be a reality in 1 or 2 years in Javascript, http://www.2ality.com/2017/01/shared-array-buffer.html). I plane to test parallelism with DataFrame-js in backend with https://github.com/turbo/js. Column-based can't be used for this purpose.
- Rows have many other advantages that I haven't in my mind right now...
However Row-based has some disadvantages:
- It's slower in some simple column manipulation cases where column-based can be better: modifiy one column, add one column, sort columns, cast one column... (but this disadvantage is canceled when you need to work on multiple columns at the same time)
- The API can be painful for simple column manipulations (mike df["mycolumn"] = df["mycolumn"] ** 2 in pandas).
- Declare a large number of Row objects is heavy and lead to slow computations...
- And a lot of other issues...
To conclude, both Column and Row have advantages (and disadvantages). I have chosen the row-based solution but it could be interesting to improve column manipulations, or to add new features. Why not create a MutableDataFrame (as scala does for some data structures) which could use similar API and column-based operations than R or pandas ? It could be interesting, but it's not in my short-time aims.
Indeed, DataFrame can be slower in some column manipulations, but it's also faster in map and reduction taks (that I use in 70% of times). I work (slowly) on a new DataFrame version including important performance (speed and memory consumption) optimizations (I hope to make it 10x faster). I will try to improve column operations and maybe to create some bridges between rows and columns (like a better .transpose() method as you said).
I hope I have answered to your question. If you have any ideas of improvment of column-based (which doesn't break the code base and the API), make a PR.
dragoljub
Hi. This code looks very interesting and exciting. :+1:
Are there any plans to implement column-based data store as the underlying data container? ie Col.js vs Row.js I'm curious what motivated you to store row-based data structures rather than column-based which are generally fixed types and could offer more compression if needed via categorical columns etc.
Probably for UI tasks you are interested in full fast row-based context access of data for tool-tips etc.
Maybe all we need is a transpose function.