Open adamklein opened 9 years ago
I couldn't help but tinker a bit - this yields a 15x speed improvement, but doesn't handle multiple columns in either the aggregate-by or group-by parameters - but it does return the group as the new index, which is nice. I'd love to know if it can be done even faster!
member frame.AggregateColumnBy(groupBy:_, aggBy:_, aggFunc:Func<_,_>) =
let dct = Dictionary<_, SeriesBuilder<_,_>>()
let gby = frame.GetColumn(groupBy)
let col = frame.GetColumn(aggBy)
let mutable i = 0
for i in [0 .. frame.RowCount-1] do
let k = gby.GetAt(i)
let v = col.GetKeyAt(i), col.GetAt(i)
match dct.TryGetValue(k) with
| true, l -> l.Add(v)
| _ -> dct.[k] <- SeriesBuilder<'TRowKey,_>()
dct.[k].Add(v)
let result = series [ for k in dct.Keys -> (k, aggFunc.Invoke(dct.[k].Series)) ]
series [(aggBy, result)] |> FrameUtils.fromColumns frame.IndexBuilder frame.VectorBuilder
And, this gets us down another 3x... I wonder if any of this can be generalized without losing speed.
member frame.FoldColumnBy(groupBy:_, aggBy:_, init:_, foldl:Func<_,_,_>) =
let dct = Dictionary<_, _>()
let gby = frame.GetColumn(groupBy)
let col = frame.GetColumn(aggBy)
let mutable i = 0
for i in [0 .. frame.RowCount-1] do
let k = gby.GetAt(i)
let v = col.GetAt(i)
match dct.TryGetValue(k) with
| true, o -> dct.[k] <- foldl.Invoke(o, v)
| _ -> dct.[k] <- foldl.Invoke(init, v)
let result = series [ for k in dct.Keys -> (k, dct.[k]) ]
series [(aggBy, result)] |> FrameUtils.fromColumns frame.IndexBuilder frame.VectorBuilder
We should have a way to do faster groupby + aggregation. Here is an example:
First, some ceremony to construct the frame:
Ok, now this is the operation that takes way too long: