freqtrade / freqtrade

Free, open source crypto trading bot
https://www.freqtrade.io
GNU General Public License v3.0
26.34k stars 5.78k forks source link

issues with donwloading data from Binance and Kraken. #3515

Closed fogkaspoker closed 4 years ago

fogkaspoker commented 4 years ago

Describe your environment

Your question

-- issue for Binance:

When I download the data for 1d timeframe I only get the last 27 Candles even though I've specified the --days command. Any ideas on why this is happening??

Also, When I use the commands, the Download is taking a lot of time and it is donwloading the trades even though it is not specified. (Like in Kraken).

I am using the command: download-data --exchange binance -t 1d --days 50 --pairs BTC/USDT

2020-06-23 22:01:15,736 - freqtrade.loggers - INFO - Verbosity set to 0 2020-06-23 22:01:15,737 - freqtrade.configuration.configuration - INFO - Using exchange binance 2020-06-23 22:01:15,816 - freqtrade.configuration.configuration - INFO - Using user-data directory: /freqtrade/user_data ... 2020-06-23 22:01:15,816 - freqtrade.configuration.configuration - INFO - Using data directory: /freqtrade/user_data/data/binance ... 2020-06-23 22:01:15,817 - freqtrade.configuration.configuration - INFO - Using pairs ['BTC/USDT'] 2020-06-23 22:01:15,817 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1d'] 2020-06-23 22:01:15,817 - freqtrade.configuration.configuration - INFO - Detected --days: 50 2020-06-23 22:01:15,817 - freqtrade.configuration.configuration - INFO - Using "json" to store OHLCV data. 2020-06-23 22:01:15,817 - freqtrade.configuration.configuration - INFO - Using "jsongz" to store trades data. 2020-06-23 22:01:15,817 - freqtrade.configuration.check_exchange - INFO - Checking exchange... 2020-06-23 22:01:15,817 - freqtrade.configuration.check_exchange - INFO - Exchange "binance" is officially supported by the Freqtrade development team. 2020-06-23 22:01:15,817 - freqtrade.configuration.config_validation - INFO - Validating configuration ... 2020-06-23 22:01:15,825 - freqtrade.commands.data_commands - INFO - About to download pairs: ['BTC/USDT'], intervals: ['1d'] to /freqtrade/user_data/data/binance 2020-06-23 22:01:15,825 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled 2020-06-23 22:01:15,825 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True} 2020-06-23 22:01:15,835 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 200} 2020-06-23 22:01:15,845 - freqtrade.exchange.exchange - INFO - Using Exchange "Binance" 2020-06-23 22:01:15,845 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Binance'... 2020-06-23 22:01:15,845 - freqtrade.exchange.exchange - WARNING - Markets were not loaded. Loading them now.. 2020-06-23 22:01:17,404 - freqtrade.data.history.history_utils - INFO - Downloading trades for pair BTC/USDT. 2020-06-23 22:01:17,405 - freqtrade.data.history.history_utils - INFO - Current Amount of trades: 0

-- issue for Kraken:

Similar issue for Kraken:

or the 1d I only get 30 days by default, then when Specified the --days 50. I still get the same 30 in the file.

--exchange kraken --pairs XRP/EUR -t 1d --days 50 --dl-trades

2020-06-23 16:29:22,150 - freqtrade.loggers - INFO - Verbosity set to 0 2020-06-23 16:29:22,150 - freqtrade.configuration.configuration - INFO - Using exchange kraken 2020-06-23 16:29:22,151 - freqtrade.configuration.configuration - INFO - Using user-data directory: /freqtrade/user_data ... 2020-06-23 16:29:22,151 - freqtrade.configuration.configuration - INFO - Using data directory: /freqtrade/user_data/data/kraken ... 2020-06-23 16:29:22,151 - freqtrade.configuration.configuration - INFO - Using pairs ['XRP/EUR'] 2020-06-23 16:29:22,151 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1d'] 2020-06-23 16:29:22,151 - freqtrade.configuration.configuration - INFO - Detected --days: 50 2020-06-23 16:29:22,152 - freqtrade.configuration.configuration - INFO - Using "json" to store OHLCV data. 2020-06-23 16:29:22,152 - freqtrade.configuration.configuration - INFO - Using "jsongz" to store trades data. 2020-06-23 16:29:22,152 - freqtrade.configuration.check_exchange - INFO - Checking exchange... 2020-06-23 16:29:22,152 - freqtrade.configuration.check_exchange - INFO - Exchange "kraken" is officially supported by the Freqtrade development team. 2020-06-23 16:29:22,152 - freqtrade.configuration.config_validation - INFO - Validating configuration ... 2020-06-23 16:29:22,159 - freqtrade.commands.data_commands - INFO - About to download pairs: ['XRP/EUR'], intervals: ['1d'] to /freqtrade/user_data/data/kraken 2020-06-23 16:29:22,159 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled 2020-06-23 16:29:22,163 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True} 2020-06-23 16:29:22,168 - freqtrade.exchange.exchange - INFO - Using Exchange "Kraken" 2020-06-23 16:29:22,168 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Kraken'... 2020-06-23 16:29:22,168 - freqtrade.exchange.exchange - WARNING - Markets were not loaded. Loading them now.. 2020-06-23 16:29:25,202 - freqtrade.data.history.history_utils - INFO - Downloading pair XRP/EUR, interval 1d. 2020-06-23 16:29:25,203 - freqtrade.data.history.history_utils - INFO - Download history data for pair: "XRP/EUR", timeframe: 1d and store in /freqtrade/user_data/data/kraken. 2020-06-23 16:29:28,409 - freqtrade.exchange.exchange - INFO - Downloaded data for XRP/EUR with length 51.

Thanks in advance for any help.

xmatthias commented 4 years ago

for binance, you should not be using --dl-trades. While it works in theory - it is rather slow and unnecessary.

You're not specifying this at the command line based on the above - however it seems like you got the download_trades key in your configuration (and set to true). It's configured like this by default for kraken - so you should verify your config.json file corresponds to the output you're expecting.

when not using --dl-trades - downloading for binance is rather fast (especially for daily candles it shouldn't take more than a few seconds).

for Kraken, it seems like it did store 51 candles ... at least that's what the log suggests.

fogkaspoker commented 4 years ago

Thanks so much for the answer. I have corrected the config.json file and now it is working very good for Binance.

but for Kraken, there is something weird:

download-data --exchange kraken -t 1d --days 50 --pairs XRP/EUR

I get the output

2020-06-24 11:25:32,854 - freqtrade.configuration.configuration - INFO - Using exchange kraken 2020-06-24 11:25:32,854 - freqtrade.configuration.configuration - INFO - Using user-data directory: /freqtrade/user_data ... 2020-06-24 11:25:32,854 - freqtrade.configuration.configuration - INFO - Using data directory: /freqtrade/user_data/data/kraken ... 2020-06-24 11:25:32,855 - freqtrade.configuration.configuration - INFO - Using pairs ['XRP/EUR'] 2020-06-24 11:25:32,855 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1d'] 2020-06-24 11:25:32,855 - freqtrade.configuration.configuration - INFO - Detected --days: 50 2020-06-24 11:25:32,855 - freqtrade.configuration.configuration - INFO - Using "json" to store OHLCV data. 2020-06-24 11:25:32,855 - freqtrade.configuration.configuration - INFO - Using "jsongz" to store trades data. 2020-06-24 11:25:32,855 - freqtrade.configuration.check_exchange - INFO - Checking exchange... 2020-06-24 11:25:32,855 - freqtrade.configuration.check_exchange - INFO - Exchange "kraken" is officially supported by the Freqtrade development team. 2020-06-24 11:25:32,856 - freqtrade.configuration.config_validation - INFO - Validating configuration ... 2020-06-24 11:25:32,863 - freqtrade.commands.data_commands - INFO - About to download pairs: ['XRP/EUR'], intervals: ['1d'] to /freqtrade/user_data/data/kraken 2020-06-24 11:25:32,863 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled 2020-06-24 11:25:32,863 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True} 2020-06-24 11:25:32,867 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 3100} 2020-06-24 11:25:32,872 - freqtrade.exchange.exchange - INFO - Using Exchange "Kraken" 2020-06-24 11:25:32,872 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Kraken'... 2020-06-24 11:25:32,872 - freqtrade.exchange.exchange - WARNING - Markets were not loaded. Loading them now.. 2020-06-24 11:25:36,025 - freqtrade.data.history.history_utils - INFO - Downloading trades for pair XRP/EUR. 2020-06-24 11:25:36,025 - freqtrade.data.history.history_utils - INFO - Current Amount of trades: 0 2020-06-24 11:32:35,363 - freqtrade.misc - INFO - dumping json to "/freqtrade/user_data/data/kraken/XRP_EUR-trades.json.gz" 2020-06-24 11:32:37,225 - freqtrade.data.history.history_utils - INFO - New Amount of trades: 133991

The file is correctly saved and I get the 50 entries.

download-data --exchange kraken -t 1d --days 70 --pairs XRP/EUR

freqtrade-uat download-data --exchange kraken -t 1d --days 70 --pairs XRP/EUR 2020-06-24 11:47:29,586 - freqtrade.loggers - INFO - Verbosity set to 0 2020-06-24 11:47:29,586 - freqtrade.configuration.configuration - INFO - Using exchange kraken 2020-06-24 11:47:29,586 - freqtrade.configuration.configuration - INFO - Using user-data directory: /freqtrade/user_data ... 2020-06-24 11:47:29,587 - freqtrade.configuration.configuration - INFO - Using data directory: /freqtrade/user_data/data/kraken ... 2020-06-24 11:47:29,587 - freqtrade.configuration.configuration - INFO - Using pairs ['XRP/EUR'] 2020-06-24 11:47:29,587 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1d'] 2020-06-24 11:47:29,587 - freqtrade.configuration.configuration - INFO - Detected --days: 70 2020-06-24 11:47:29,587 - freqtrade.configuration.configuration - INFO - Using "json" to store OHLCV data. 2020-06-24 11:47:29,587 - freqtrade.configuration.configuration - INFO - Using "jsongz" to store trades data. 2020-06-24 11:47:29,587 - freqtrade.configuration.check_exchange - INFO - Checking exchange... 2020-06-24 11:47:29,587 - freqtrade.configuration.check_exchange - INFO - Exchange "kraken" is officially supported by the Freqtrade development team. 2020-06-24 11:47:29,588 - freqtrade.configuration.config_validation - INFO - Validating configuration ... 2020-06-24 11:47:29,594 - freqtrade.commands.data_commands - INFO - About to download pairs: ['XRP/EUR'], intervals: ['1d'] to /freqtrade/user_data/data/kraken 2020-06-24 11:47:29,595 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled 2020-06-24 11:47:29,595 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True} 2020-06-24 11:47:29,599 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 3100} 2020-06-24 11:47:29,603 - freqtrade.exchange.exchange - INFO - Using Exchange "Kraken" 2020-06-24 11:47:29,604 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Kraken'... 2020-06-24 11:47:29,604 - freqtrade.exchange.exchange - WARNING - Markets were not loaded. Loading them now.. 2020-06-24 11:47:32,757 - freqtrade.data.history.history_utils - INFO - Downloading trades for pair XRP/EUR. 2020-06-24 11:47:33,326 - freqtrade.data.history.history_utils - INFO - Using last trade date -5s - Downloading trades for XRP/EUR since: 2020-06-24T11:41:12. 2020-06-24 11:47:33,326 - freqtrade.data.history.history_utils - INFO - Current Amount of trades: 134029 2020-06-24 11:47:39,790 - freqtrade.misc - INFO - dumping json to "/freqtrade/user_data/data/kraken/XRP_EUR-trades.json.gz" 2020-06-24 11:47:41,666 - freqtrade.data.history.history_utils - INFO - New Amount of trades: 134037

I tried the same with binance and it works fine, the file gets updated with the new entries. Do we need to gather all the data at once with kraken?

Thanks in advance,

xmatthias commented 4 years ago

Well the intend with download-data is to download the longest period - and then do incremental steps to bridge the gap between the last data and now.

Downloading data "before" the data does work for ohlcv - but by removing all data you got. Apparently there's a bug in the "trades" implementation - but it would "drop" all your trades anyway - so just start with the longest timeperiod you'll need (say 200 days) - it'll take ~forever~ a bit to download (assuming you use multiple pairs) - and then you run incremental catchups every few days (which won't take this long, if done regularily).

fogkaspoker commented 4 years ago

I understand much better now, so it is intented to be used the other way around. First download as much data as possible then updating with the latest values.

Thanks so much for the clarification. And for the great support!

xmatthias commented 4 years ago

we'll still also support increasing the range - but by redownloading all of the data .. which obviously will be kinda slow.

fogkaspoker commented 4 years ago

I see, so the bot actually uses the --dl-trades to build the 1d 1h, etc files this means, if I extend the data range and download the data again. I would need to erase de timeframe file and re-generate it. I think I've got it.

Thanks

xmatthias commented 4 years ago

no, the timeframe files (5m, 1d, ...) are generated automatically. it's just the trades files that are currently not deleted - but #3519 will fix that.