VersionStore - list_symbols and snapshot slows down where there are a lot of version documents

man-group / arctic

High performance datastore for time series and tick data

https://arctic.readthedocs.io/en/latest/

GNU Lesser General Public License v2.1

3.06k stars 583 forks source link

VersionStore - list_symbols and snapshot slows down where there are a lot of version documents #855

Closed rob256 closed 3 years ago

rob256 commented 4 years ago

Arctic Version

1.179.3

Arctic Store

VersionStore

Platform and version

CentOS 7

Description of problem and/or code sample that reproduces the issue

list_symbols performs a COLLSCAN on the .versions collections. This isn't too bad when the collection is small, but when there are 100,000 or millions of documents then the slowdown is noticeable.

shashank88 commented 3 years ago

@rob256 did we merge this one?

rob256 commented 3 years ago

@shashank88 we haven't merged this one yet as the PR was failing on tests due to pandas version. I'm happy for this to be merged whenever though!