Chunck store read is very slow

man-group / arctic

High performance datastore for time series and tick data

https://arctic.readthedocs.io/en/latest/

GNU Lesser General Public License v2.1

3.05k stars 583 forks source link

Chunck store read is very slow #900

Closed Harishangaran closed 2 years ago

Harishangaran commented 3 years ago

Arctic Version

# 1.80

Arctic Store

# ChunkStore

Platform and version

Very slow read query take 77s to read 4000 rows of daily close price compared to same data in version store takes 0.09s.

shashank88 commented 3 years ago

@Harishangaran can you put the repro steps here? 77s seems way off.

Harishangaran commented 3 years ago

Hi Shashank,

The pandas version is 1.0.3

########################################## from connect.mongoConnect import * from timerUtil.timer import Timer

store = Arctic(client)

library = store['test']

######## READ PRICES #################### t = Timer() t.start() item = library.read('GOOG') t.stop()

print(item)

Harishangaran commented 3 years ago

Sample data

Index Close 2004-09-02 49.4098014831543 2004-08-19 49.813289642333984 2004-09-08 50.181907653808594 2004-09-03 50.28651428222656 2004-08-20 50.316402435302734 2004-09-07 50.316402435302734

Harishangaran commented 3 years ago

Any idea why it could be?

bmoscon commented 3 years ago

its either your version of pandas or its been recently fixed by the PR that @TomTaylorLondon merged in

jasonlocal commented 2 years ago

Legacy issue - might have been resolved by the latest version. Please feel free to reopen this ticket if this is still an issue