ranaroussi / pystore

Fast data store for Pandas time-series data
Apache License 2.0
562 stars 101 forks source link

Append not working #3

Closed trbck closed 6 years ago

trbck commented 6 years ago

Using latest version 0.0.12 the append function does not seem to work nor raises an error

Initial dataframe's last row:

df.tail(2)

Close High Low Open Volume 
2017-12-29 00:00:00+00:00 167.15 168.65 167.05 168.25 364619 
2018-01-02 00:00:00+00:00 167.15 167.7 165.25 167.15 587067 

Saving and appending:

store = pystore.store('mydatastore') 
collection = store.collection('eod') 
last = df[-1:] 
collection.write('symbol', df[:-1], overwrite=True) 
collection.append('symbol', last) 
df = collection.item('symbol').to_pandas() 
df.tail(1) 

yields:

Close High Low Open Volume index 
2017-12-29 167.15 168.65 167.05 168.25 364619 

At least on my end. What am I doing wrong here?

ranaroussi commented 6 years ago

Try using df.tail() instead of df.tail(1) which only displays the last record.

trbck commented 6 years ago

I know, just wanted to simply highlight that it does not seem to append the last row :)

trbck commented 6 years ago

Also the length of the df stays the same. Might this somehow be related to snappy?

ranaroussi commented 6 years ago

I can't reproduce this on any of my machines...

Try:

store = pystore.store('mydatastore') 
collection = store.collection('eod') 

# write
collection.write('symbol', df[:-1], overwrite=True) 

# append
collection.append('symbol',  df[-1:] ) 

# read
collection.item('symbol').to_pandas() .tail() 
trbck commented 6 years ago

Got it. The time offset in the dt index was the reason the append did not work an my end. It works when I strip it from the dt value ("2018-07-13 20:03:00" not "2018-07-13 20:03:00+00:00"). Thanks again!