search

stefan-jansen / machine-learning-for-trading

Code for Machine Learning for Algorithmic Trading, 2nd edition.
https://ml4trading.io
13.3k stars 4.2k forks source link

Where are 'itch.h5' and 'order_book.h5' files? #178

Closed emskiphoto closed 3 years ago

emskiphoto commented 3 years ago

Where are the 'itch.h5' and 'order_book.h5' files that are used in this notebook:

https://github.com/PacktPublishing/Machine-Learning-for-Algorithmic-Trading-Second-Edition/blob/master/02_market_and_fundamental_data/01_NASDAQ_TotalView-ITCH_Order_Book/01_parse_itch_order_flow_messages.ipynb

MBounouar commented 3 years ago

You just need first to follow the cells. The files will be downloaded from ftp://emi.nasdaq.com/ITCH and then parsed. The process is very slow as several gigabytes of data need to be downloaded and the parsing and storing to h5 files also takes time because of the of the way that the itch data is stored in the binary

emskiphoto commented 3 years ago

Thank you. I have a new problem, when I attempt to "processes the binary file and produces the parsed orders stored by message type" this error occurs:

HDF5ExtError: HDF5 error back trace

File "D:\pytables_hdf5\CMake-hdf5-1.10.5\hdf5-1.10.5\src\H5G.c", line 683, in H5Gclose not a group

End of HDF5 error back trace

Problems closing the Group stock_locate

stefan-jansen commented 3 years ago

This suggests the HDF5 file got somehow corrupted while processing the data. I would delete the HDF5 file and re-run the download.

Closing for now but please feel free to reopen if you run into this issue again while processing..

emskiphoto commented 3 years ago

Deleting the previous HDF5 file and re-running the download resolved the issue - thank you! FYI, It took about 51 minutes to parse the orders by message type on a 16GB i7 Win10 laptop.