K2InformaticsGmbH / dderl

Web based Data Explorer / Data Jump Station with Erlang In-Memory Support
Apache License 2.0
23 stars 12 forks source link

Distinct count takes too long for big tables. #613

Open acautin opened 5 years ago

acautin commented 5 years ago

It seems most of the time is spent finding if the values are already present here: https://github.com/K2InformaticsGmbH/dderl/blob/master/src/dderl_fsm.erl#L1559

Evaluate if the performance can be improved by using ets tables or maps for accumulating the intermediate results without having a big impact in the memory consumption/garbage generated.