search

man-group / arctic

High performance datastore for time series and tick data
https://arctic.readthedocs.io/en/latest/
GNU Lesser General Public License v2.1
3.05k stars 584 forks source link

Arctic library (mongodb collections) - Best way to store 1000's of symbols for 15 years of history #891

Open DataCT2020 opened 3 years ago

DataCT2020 commented 3 years ago

Arctic Version

# Arctic version here

Arctic Store


# VersionStore, TickStore, or ChunkStore
```ChunkStore

#### Platform and version

Put here

#### Description of problem and/or code sample that reproduces the issue

Hi
We are using Arctic library data to process trades/quotes historical data. For these data we have 1000's of symbols and we are expecting to load last 15 years of history. So far we have loaded 1 year of history and everything is working as expected. 

We are using Arctic Chunkstore with Date Chunker to store these data.

We would like to know is it ok to store all data in one collection (arctic library)? or should we split collections by based on date range of data ? (e.g 2017 to 2020 data in collection1? 2013 to 2016 data in collection2).

Thanks for your time

Dhru
shashank88 commented 3 years ago

I will let @bmoscon reply on the Chunkstore stuff, but it should be fine to store multi-year history in one collection if it's sharded properly (if you do have shards) and the indexes in Mongo look okay which most of the stores create for you.

DataCT2020 commented 3 years ago

thank for reply shashank and I will wait for further confirmation from @bmoscon