search

JuliaData / DataTables.jl

(DEPRECATED) A rewrite of DataFrames.jl based on Nullable
Other
29 stars 11 forks source link

Specialize row_group_slots() and findrow() on column types to improve performance #79

Closed nalimilan closed 7 years ago

nalimilan commented 7 years ago

Looping over columns is very slow when their type is unknown at compile time. Specialize the method on the types of the key (grouping) columns by passing a tuple of columns rather than a DataTable. This will force compiling a specific method for each combination of key types, but their number should remain relatively low and the one-time cost is worth it.

This dramatically improves performance of groupby(), but does not have a large effect on join() since it is very inefficient in other areas.

This is an alternative to https://github.com/JuliaData/DataTables.jl/pull/76. We should probably merge it anyway, even if we merge https://github.com/JuliaData/DataTables.jl/pull/76 later, as it improves the performance of the current algorithm, making it possible to compare the merits of both approaches.

In the simple test used in #76, this makes grouping faster than DataFrames, and even slightly faster than #76. More benchmarks would be needed to check whether this is also the case for other scenarios. Unfortunately, join remains much slower than DataFrames for unrelated reasons.

The downside of this change is that the method will be recompiled for each combination of column types used for grouping. I think that's OK since the number of grouping columns is generally low, and the number of different types is limited too. Though I should note the approach in #76 only compiles one method for each column types, not for each of their combinations.

Cc: @cjprybol @alyst

nalimilan commented 7 years ago

Looks like I spoke too soon about join. Using the benchmark from https://github.com/JuliaStats/DataFrames.jl/pull/850 adapted to DataTables, I get a 3x performance improvement compared with master and with DataFrames. We need more systematic benchmarks to check in what cases this applies.

EDIT: also, using the benchmark from this Discourse post, performance is similar to DataFrames and only slightly worse than Pandas, when it was previously terrible.

aaowens commented 7 years ago

The by performance is pretty close to Pandas with the Discourse example. On joins, Pandas is 2x faster and R's data.table is 3-4x faster.