search

edtechre / pybroker

Algorithmic Trading in Python with Machine Learning
https://www.pybroker.com
Other
2.09k stars 261 forks source link

Data caching issue #93

Closed ASDWQad closed 8 months ago

ASDWQad commented 10 months ago

I have customized a Binance data source, but the API can only retrieve 1,000 data points each time. I have cached all the data from 2023/1/1 to 2024/1/1, but when calling the cache, it does not merge the kline return, only returning the cached 1,000 data points.

Can this problem be solved? Or I can only download the data to the local, and then customize the data source to import the local data.

图片
edtechre commented 10 months ago

Hi @ASDWQad,

You should be able to reuse the existing cache mechanism by extending DataSourceCacheMixin. Can you post your code here?

tsunamilx commented 8 months ago

Hi @edtechre

I have another question about the cache: When I query for some small amount of symbols (like 100), the cache works, but when I query for large amount of symbols (like 1000), it always calls the fetch function (meaning cache is not used). Is there any limit on the cache (like max size limit) so that the early data is being evicted?

And when I query the 1000 symbols one by one, it actually loads everything from cache. 

# querying 1000 symbols at once fails the cache
d = ds.query(symbols, start, end, timeframe)

# querying 1000 symbols one by one works
for s in symobls:
  d = ds.query(s, start, end, timeframe)
edtechre commented 8 months ago

Thanks @tsunamilx, I am not aware of any cache limit. I will need to test this myself and report back.

edtechre commented 8 months ago

Hi @tsunamilx,

Looking at the code, there is no difference between calling query with all symbols or individually for each symbol. See data.py.

Perhaps your cache was invalidated somehow?