visualfabriq / bquery

A query and aggregation framework for Bcolz (W2013-01)
https://www.visualfabriq.com
BSD 3-Clause "New" or "Revised" License
56 stars 11 forks source link

reallocate array if needed #36

Closed FrancescElies closed 9 years ago

FrancescElies commented 9 years ago

https://github.com/visualfabriq/bquery/issues/35

FrancescElies commented 9 years ago

https://gist.github.com/FrancescElies/ad48321b6fb314bbaf5a before (commit https://github.com/visualfabriq/bquery/commit/82dd28d35c61b6bdf6f4f7d8fd46fb7720032d4c)

$ python ~/westland_basket_analysis_v2.py
0.3261 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.3761 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.3245 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.4529 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.3675 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.4587 sec

now (commit https://github.com/visualfabriq/bquery/commit/0ecaaa6356c4c00f3ce650f961bf41937d8ad15b)

$ python ~/westland_basket_analysis_v2.py
0.1668 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.2298 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.1975 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.1448 sec
(venv)
$ python ~/westland_basket_analysis_v2.py
0.2154 sec
FrancescElies commented 9 years ago

Added small benchmark

before (commit 82dd28d): is_in_ordered_subgroups: 0.8003 sec is_in_ordered_subgroups: 0.763 sec is_in_ordered_subgroups: 0.7526 sec

afterwards (commit 219dda34bbc453732b88cb0a0dc98e96d262a35e): is_in_ordered_subgroups: 0.4617 sec is_in_ordered_subgroups: 0.2779 sec is_in_ordered_subgroups: 0.311 sec

FrancescElies commented 9 years ago

One million records improved x2 (old) is_in_ordered_subgroups: 4.6336 sec (new) is_in_ordered_subgroups: 2.1973 sec

FrancescElies commented 9 years ago

@CarstVaartjes: just to let you know, merging & updating our servers