Make Group Index carray based & cached

visualfabriq / bquery

A query and aggregation framework for Bcolz (W2013-01)

https://www.visualfabriq.com

BSD 3-Clause "New" or "Revised" License

56 stars 11 forks source link

Closed CarstVaartjes closed 8 years ago

CarstVaartjes commented 8 years ago

To do to improve the Group Index performance (used by aggregations of multiple columns):

Make creating the group index carray based instead of numpy (which will greatly improve memory usage)
Cache multi-column unique combinations, which will greatly enhance the speed

This will greatly reduce the memory usage and speed of complex queries

CarstVaartjes commented 8 years ago

Also related to #44

CarstVaartjes commented 8 years ago

Was handled in #73