Could not convert XX/XX to OHLCV

[mmintel]

Describe your environment

• Operating system: Windows with WSL2
• Python Version: 3.9.1 (python -V)
• CCXT version: 1.40.99 (pip freeze | grep ccxt)
• Freqtrade Version: freqtrade develop-5ab8cc56 (freqtrade -V or docker-compose run --rm freqtrade -V for Freqtrade running in docker)
• I am not running freqtrade in docker as it leads to problems with permissions with root

Describe the problem:

After downloading data from Kraken I waited for around a day and then it crashed or skipped most of the pairs.

Steps to reproduce:

1. freqtrade download-data --exchange kraken --timerange 20200601-20210101 --timeframes 1m 5m 1h -v --dl-trades

Observed Results:

• data download skipped few pairs .e.g. LINK/EUR, LTC/EUR... but I expected it to save everything into the data folder
• instead just others like AAVE/EUR, DOT/EUR, BTC/EUR ... were saved successfully

Relevant code exceptions or logs

2021-01-26 14:21:05,094 - freqtrade.data.history.history_utils - ERROR - Could not convert ETH/EUR to OHLCV.
Traceback (most recent call last):
  File "/home/mmintel/Projects/Private/freqtrade/freqtrade/data/history/history_utils.py", line 358, in convert_trades_to_ohlcv
    ohlcv = trades_to_ohlcv(trades, timeframe)
  File "/home/mmintel/Projects/Private/freqtrade/freqtrade/data/converter.py", line 191, in trades_to_ohlcv
    raise ValueError('Trade-list empty.')
ValueError: Trade-list empty.
2021-01-26 14:21:05,098 - freqtrade.data.history.history_utils - ERROR - Could not convert ETH/EUR to OHLCV.
Traceback (most recent call last):
  File "/home/mmintel/Projects/Private/freqtrade/freqtrade/data/history/history_utils.py", line 358, in convert_trades_to_ohlcv
    ohlcv = trades_to_ohlcv(trades, timeframe)
  File "/home/mmintel/Projects/Private/freqtrade/freqtrade/data/converter.py", line 191, in trades_to_ohlcv
    raise ValueError('Trade-list empty.')
ValueError: Trade-list empty.
2021-01-26 14:21:05,098 - freqtrade.data.history.history_utils - ERROR - Could not convert ETH/EUR to OHLCV.
Traceback (most recent call last):
  File "/home/mmintel/Projects/Private/freqtrade/freqtrade/data/history/history_utils.py", line 358, in convert_trades_to_ohlcv
    ohlcv = trades_to_ohlcv(trades, timeframe)
  File "/home/mmintel/Projects/Private/freqtrade/freqtrade/data/converter.py", line 191, in trades_to_ohlcv
    raise ValueError('Trade-list empty.')
ValueError: Trade-list empty.

Additional notes

I've also found this issue here https://github.com/freqtrade/freqtrade/pull/3971 but it seems the PR containing a potential fix doesn't address this one here.

[xmatthias]

3971 does address this, prior to that PR it was crashing if ONE pair failed to download (or if a wrong pair was given).

Unfortunately, your logs don't really contain the relevant things. The conversation fails because no trade data is available. you'll notice that for example for ETH/EUR, there's no *-trades.json.gz in the data folder (usually user_data/data/kraken).

The way data-download works for kraken (downloading trades) is as follows:

• download trades data (takes forever and a bit due to kraken's rate limits)
• convert trades to ohlcv (this is where your log starts).

Unfortunately, the log above starts with the conversion - the "real" problem / error will have happened during download. Without that part of the log - it'll be difficult to know why data for ETH/EUR was not downloaded.

[mmintel]

Unfortunately I don't have these logs anymore. If I understand you correctly the process wasn't able to create the data folder? Could be a permissions problem then?

I've tried reproducing this with a lower amount of trades and only one pair but that was saved succesfully. Guess I just have to run the command mentioned above again and see if I can reproduce it then.

[xmatthias]

if it worked for one pair, then permissions (on the folder level) are correct.

It can fail to write the data ... but it can also fail downloading the data - maybe because of a network "hiccup" - or an exchange API hiccup. krakens status page doesn't indicate this in the last week so it's pretty unlikely (unless it was limited to one geographical region)

The logs would tell where / what failed ... but i know the logs for downloading a lot of data can become verbose - so still having them is a question of luck if you didn't use --logfile datadownload.log ...

[mmintel]

Ok good point then I will save the logfile next time and post it here.

[mmintel]

I was running this again on a lower timeframe on my windows as well as on my mac and it worked out. For anybody else having this problem: Downloading from kraken takes ages so I recommend download the data instead and converting them. Going to close this now as I am not going to run this for days again :D