search

FX31337 / FX-BT-Scripts

:page_facing_up: Useful scripts for backtesting.
MIT License
35 stars 39 forks source link

dl_bt_dukascopy.py: EOFError: Compressed file ended before the end-of-stream marker was reached #3

Open kenorb opened 9 years ago

kenorb commented 9 years ago

Command:

./dl_bt_dukascopy.py -y 2013 -p EURUSD -c

Traceback:

Converting into CSV (download/EURUSD/2013-01-07--01h_ticks.csv)...
Traceback (most recent call last):
  File "./dl_bt_dukascopy.py", line 159, in <module>
    ds.bt5_to_csv()
  File "./dl_bt_dukascopy.py", line 102, in bt5_to_csv
    data = f.read()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lzma.py", line 310, in read
    return self._read_all()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lzma.py", line 251, in _read_all
    while self._fill_buffer():
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lzma.py", line 225, in _fill_buffer
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

It happens randomly. In this case, it happens for: 2013-01-07--01h_ticks.bi5 specifically.

ghost commented 9 years ago

https://github.com/d-a-t-a-d-i-t/FX-BT-Scripts/commit/d119202a17db91351b29489505b99330039206f3

kenorb commented 9 years ago

Fixed and merged.

kenorb commented 8 years ago

Still there is a problem:

./dl_bt_dukascopy.py -c -p AUDJPY -y 2007 -m 10
Downloading http://www.dukascopy.com/datafeed/AUDJPY/2007/09/25/02h_ticks.bi5 into: download/dukascopy/AUDJPY/2007/10/2007-10-25--02h_ticks.bi5...
File (download/dukascopy/AUDJPY/2007/10/2007-10-25--02h_ticks.bi5) exists, so skipping.
Converting into CSV (download/dukascopy/AUDJPY/2007/10/2007-10-25--02h_ticks.csv)...
Traceback (most recent call last):
  File "./dl_bt_dukascopy.py", line 169, in <module>
    ds.bt5_to_csv()
  File "./dl_bt_dukascopy.py", line 110, in bt5_to_csv
    data = f.read()
  File "/usr/local/lib/python3.2/dist-packages/backports/lzma/__init__.py", line 287, in read
    return self._read_all()
  File "/usr/local/lib/python3.2/dist-packages/backports/lzma/__init__.py", line 236, in _read_all
    while self._fill_buffer():
  File "/usr/local/lib/python3.2/dist-packages/backports/lzma/__init__.py", line 223, in _fill_buffer
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
$ python3 --version
Python 3.2.3

Any clues?

ghost commented 8 years ago

As I experienced, the ideal lzma._BUFFER_SIZE value (which doesn't cause EOFError exception) varies from Python version to Python version, so it's different for e.g. Python 3.2.x and 3.4.x. The solutions could be:

  1. try different values with different Python versions, or
  2. create a fix/workaround in upstream Python code.
ghost commented 8 years ago

or

  1. call lzma decompressor externally from the script until LZMA API is fixed in Python.
kenorb commented 8 years ago

PR: #35

kenorb commented 8 years ago

Possible workaround: