Open mkborregaard opened 7 years ago
Are you imagining that colwise!
would work like an in-place aggregate?
example:
julia> dt
10×2 DataTables.DataTable
│ Row │ A │ B │
├─────┼────┼────┤
│ 1 │ 1 │ 1 │
│ 2 │ 2 │ 2 │
│ 3 │ 3 │ 3 │
│ 4 │ 4 │ 4 │
│ 5 │ 5 │ 5 │
│ 6 │ 6 │ 6 │
│ 7 │ 7 │ 7 │
│ 8 │ 8 │ 8 │
│ 9 │ 9 │ 9 │
│ 10 │ 10 │ 10 │
julia> colwise!(normalize, dt)
10×2 DataTables.DataTable
│ Row │ A │ B │
├─────┼───────────┼───────────┤
│ 1 │ 0.0509647 │ 0.0509647 │
│ 2 │ 0.101929 │ 0.101929 │
│ 3 │ 0.152894 │ 0.152894 │
│ 4 │ 0.203859 │ 0.203859 │
│ 5 │ 0.254824 │ 0.254824 │
│ 6 │ 0.305788 │ 0.305788 │
│ 7 │ 0.356753 │ 0.356753 │
│ 8 │ 0.407718 │ 0.407718 │
│ 9 │ 0.458682 │ 0.458682 │
│ 10 │ 0.509647 │ 0.509647 │
# compared to
julia> aggregate(dt, normalize)
10×2 DataTables.DataTable
│ Row │ A_normalize │ B_normalize │
├─────┼─────────────┼─────────────┤
│ 1 │ 0.0509647 │ 0.0509647 │
│ 2 │ 0.101929 │ 0.101929 │
│ 3 │ 0.152894 │ 0.152894 │
│ 4 │ 0.203859 │ 0.203859 │
│ 5 │ 0.254824 │ 0.254824 │
│ 6 │ 0.305788 │ 0.305788 │
│ 7 │ 0.356753 │ 0.356753 │
│ 8 │ 0.407718 │ 0.407718 │
│ 9 │ 0.458682 │ 0.458682 │
│ 10 │ 0.509647 │ 0.509647 │
That should be pretty straight forward to do. What would be the preferable behavior for functions that reduce to scalars like sum/length or extrema? Fail on a dimension mismatch or resize the DataTable to be a single row? (I'm not sure if resize!(column, 1)
will work, in which case it'd have to throw an error because it couldn't modify in place).
example
julia> colwise!(length, dt)
1×2 DataTables.DataTable
│ Row │ A │ B │
├─────┼────┼────┤
│ 1 │ 10 │ 10 │
# or
Error: DimensionMismatch: columns cannot be resized in place, use aggregate(function, dt) instead
aggregate
is made to be used with a grouping variable. colwise!
would just be an in-place equivalent of colwise
, so something very simple. Cf. https://discourse.julialang.org/t/announcement-dataframes-future-plans/266/25.
Though thinking about it more, I'm not sure how useful such a function would be. colwise
is mainly useful for summary functions which return one or two values for each column. OTC, colwise!
would only work for functions which return the vectors or the same length as the input. So probably better give this function a different name.
The other issue with such a function is that colwise!(col -> col .- mean(col), df)
it would encourage an inefficient pattern which allocates a copy of col
before replacing the old column. A manual loop with x0 = mean(col); map!(x -> x - x0, col)
is much more efficient in that regard. So if we offer such a helper function, it should be smarter than that. We could have it call the function for each element of a column, with additional column-wise arguments à la broadcast
. For example: colwise!((x, x0) -> x - x0, df, colwise(mean, df)
.
Reading your concerns here and thinking about it more, I am also not sure this functionality would be broadly useful. It came out of a natural impulse to work with a DataTable inplace because this is so ideomatic in julia.
Maybe if map! could simply take an eachcol
iterator as argument?
Yes, that would make sense. Maybe a bit surprising that you would be able to modify an iterator, but doesn't sound like a big issue.
Leaving a breadcrumb to https://stackoverflow.com/questions/44235201/shortcut-to-transforming-a-dataframe, which provides an example of a use case for this.
DataTables implement a
colwise
method, but nocolwise!
. There aren't a massive number of use cases for this, but I can think of e.g. data centering and normalization. If there are no major technical obstacles with making such a function I think it'd make a nice addition.