search

cuemacro / findatapy

Python library to download market data via Bloomberg, Eikon, Quandl, Yahoo etc.
Apache License 2.0
1.69k stars 196 forks source link

Can't pull any crypto data #32

Closed NSSmithPortfolio closed 4 years ago

NSSmithPortfolio commented 4 years ago

Just using the examples, trying a few different data sources:

from findatapy.market import Market, MarketDataRequest, MarketDataGenerator

market = Market(market_data_generator=MarketDataGenerator())

md_request = MarketDataRequest(start_date='01 Jan 2018', finish_date='01 Feb 2018', cut='LOC', freq='intraday', data_source='gdax', category='crypto', fields=['close', 'volume', 'low', 'high'], tickers=['XBTUSD'])

df = market.fetch_market(md_request) print(df.head(5)) print(df.tail(5))

Gets this:

D:\CC\pandastest\venv\Scripts\python.exe D:/CC/pandastest/main.py Traceback (most recent call last): File "D:/CC/pandastest/main.py", line 10, in df = market.fetch_market(md_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\market.py", line 182, in fetch_market data_frame = self.market_data_generator.fetch_market_data(md_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 185, in fetch_market_data data_frame_agg = self.download_intraday_tick(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 304, in download_intraday_tick data_frame_single = self.fetch_single_time_series(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 403, in fetch_single_time_series data_frame_single = self.get_data_vendor(market_data_request.data_source).load_ticker(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\datavendorweb.py", line 1054, in load_ticker json_url = gdax_url.format(market_data_request_vendor.tickers[0], start_time.isoformat(), data_end_time.isoformat(), period) TypeError: 'NoneType' object is not subscriptable 2020-09-30 06:33:13,284 - findatapy.market.datavendor - ERROR - Couldn't find ticker conversion, did you type it correctly: XBTUSD 2020-09-30 06:33:13,284 - findatapy.market.datavendor - WARNING - Couldn't find field conversion, did you type it correctly: close 2020-09-30 06:33:13,284 - findatapy.market.datavendorweb - INFO - Request data from Gdax.

saeedamen commented 4 years ago

Thanks for flagging @NSSmithPortfolio. I've made some changes which I hope should fix it.

Some of the ticker conversions had been in the CSV files (for converting back and forth between findatapy format/vendor format). Also, since writing the code, it looks like many of the vendors have reduced the amount of data you can download with one call, so you might need to try smaller periods to download (eg. for GDAX/Coinbase Pro, it looks like 300 points, ie. 5 hours of 1 min data/intraday data).

NSSmithPortfolio commented 4 years ago

Hey, thanks for taking a look. I uninstalled and reinstalled in Pycharm, if that refreshes files. Changed your example to be BTCUSD and about 2 weeks of daily prices.

Now when I run this: from findatapy.market import Market, MarketDataRequest, MarketDataGenerator

market = Market(market_data_generator=MarketDataGenerator())

md_request = MarketDataRequest(start_date='07 Feb 2020', finish_date='20 Feb 2020', cut='LOC', freq='daily', data_source='binance', category='crypto', fields=['close', 'volume', 'quote-asset-volume'], tickers=['BTCUSD'])

df = market.fetch_market(md_request) print(df.head(5)) print(df.tail(5))

Gets this: Traceback (most recent call last): File "D:/CC/pandastest/main.py", line 10, in df = market.fetch_market(md_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\market.py", line 182, in fetch_market data_frame = self.market_data_generator.fetch_market_data(md_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 190, in fetch_market_data data_frame_agg = self.download_daily(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 529, in download_daily data_frame_agg = self.fetch_group_time_series(market_data_request_list) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 435, in fetch_group_time_series data_frame_group = result.get() File "C:\Users\N\Anaconda3\lib\multiprocessing\pool.py", line 644, in get raise self._value File "C:\Users\N\Anaconda3\lib\multiprocessing\pool.py", line 119, in worker result = (True, func(*args, *kwds)) File "C:\Users\N\Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar return list(map(args)) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 403, in fetch_single_time_series data_frame_single = self.get_data_vendor(market_data_request.data_source).load_ticker(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\datavendorweb.py", line 885, in load_ticker json_url = binance_url.format(market_data_request_vendor.tickers[0],period,start_time,finish_time) TypeError: 'NoneType' object is not subscriptable 2020-10-01 12:57:19,072 - findatapy.market.datavendor - ERROR - Couldn't find ticker conversion, did you type it correctly: BTCUSD 2020-10-01 12:57:19,072 - findatapy.market.datavendor - WARNING - Couldn't find field conversion, did you type it correctly: close 2020-10-01 12:57:19,072 - findatapy.market.datavendorweb - INFO - Request data from Binance

Process finished with exit code 1

saeedamen commented 4 years ago

I don't think there's a symbol 'BTCUSD' on binance. I did try 'BTCUSDT' though and that works. You won't need to reinstall PyCharm, you just need to redownload findatapy again from GitHub. I've also made a new release on PyPI with these changes, if you want to use pip to install findatapy instead.

NSSmithPortfolio commented 4 years ago

Yea, I uninstalled it with pip and now reinstalled with pip. Ran above code with BTCUSDT, no dice. Hmmm

saeedamen commented 4 years ago

Apologies, just checked the symbols, it should be XBTUSDT rather than BTCUSDT.

NSSmithPortfolio commented 4 years ago

from findatapy.market import Market, MarketDataRequest, MarketDataGenerator

market = Market(market_data_generator=MarketDataGenerator())

md_request = MarketDataRequest(start_date='07 Feb 2020', finish_date='20 Feb 2020', cut='LOC', freq='daily', data_source='binance', category='crypto', fields=['close', 'volume', 'quote-asset-volume'], tickers=['XBTUSDT'])

df = market.fetch_market(md_request) print(df.head(5)) print(df.tail(5))

2020-10-01 18:56:25,687 - findatapy.market.datavendor - ERROR - Couldn't find ticker conversion, did you type it correctly: XBTUSDT 2020-10-01 18:56:25,689 - findatapy.market.datavendor - WARNING - Couldn't find field conversion, did you type it correctly: close 2020-10-01 18:56:25,689 - findatapy.market.datavendorweb - INFO - Request data from Binance Traceback (most recent call last): File "D:/CC/pandastest/main.py", line 10, in df = market.fetch_market(md_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\market.py", line 182, in fetch_market data_frame = self.market_data_generator.fetch_market_data(md_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 190, in fetch_market_data data_frame_agg = self.download_daily(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 529, in download_daily data_frame_agg = self.fetch_group_time_series(market_data_request_list) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 435, in fetch_group_time_series data_frame_group = result.get() File "C:\Users\N\Anaconda3\lib\multiprocessing\pool.py", line 644, in get raise self._value File "C:\Users\N\Anaconda3\lib\multiprocessing\pool.py", line 119, in worker result = (True, func(*args, *kwds)) File "C:\Users\N\Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar return list(map(args)) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\marketdatagenerator.py", line 403, in fetch_single_time_series data_frame_single = self.get_data_vendor(market_data_request.data_source).load_ticker(market_data_request) File "D:\CC\pandastest\venv\lib\site-packages\findatapy\market\datavendorweb.py", line 885, in load_ticker json_url = binance_url.format(market_data_request_vendor.tickers[0],period,start_time,finish_time) TypeError: 'NoneType' object is not subscriptable

saeedamen commented 4 years ago

I get this output below, by copy & pasting that code. You might still be using the old version of the code (pip could have been using a cached version of the library)? Doing pip install findatapy==0.1.18 should hopefully pull the latest version.

2020-10-02 07:39:26,896 - findatapy.market.datavendorweb - INFO - Request data from Binance 2020-10-02 07:39:29,193 - numexpr.utils - INFO - Note: NumExpr detected 28 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-10-02 07:39:29,193 - numexpr.utils - INFO - NumExpr defaulting to 8 threads. 2020-10-02 07:39:29,197 - findatapy.market.datavendorweb - INFO - Completed request from Binance ....

        XBTUSDT.close  XBTUSDT.volume  XBTUSDT.quote-asset-volume

Date
2020-02-07 9813.730469 43966.113281 430744864.0 2020-02-08 9895.049805 43600.843750 427781792.0 2020-02-09 10151.750000 43408.476562 437441696.0 2020-02-10 9851.830078 59573.085938 590091328.0

NSSmithPortfolio commented 4 years ago

Recreated a new virtual environment, reinstalled, all good now! Thanks for working on this!