betcode-org / betfair

betfairlightweight - Betfair API-NG python wrapper (with streaming)
MIT License
418 stars 146 forks source link

add bz2 decompression option for historical stream generator #483

Closed gos-qs closed 2 years ago

gos-qs commented 2 years ago

For my use it is useful to be able to read the historical bz2 files directly into the historical stream generator rather than having to uncompress a file back to disk and then re-read it back in as raw text, and I couldn't find any existing way to do that, so I added HistoricalGeneratorStreamBz2 that does. Is this useful to anyone else and to include in the original? and if so, is the way i've implemented it ok? I figured the alternative would be to put the if-else in HistoricalGeneratorStream._read_loop, but it looked more straightforward to include a manual parameter, compression: str, to indicate files compressed but without an extension in the former method.

modified: betfairlightweight/endpoints/streaming.py modified: betfairlightweight/streaming/init.py modified: betfairlightweight/streaming/betfairstream.py

Thanks

liampauling commented 2 years ago

We recommend using smart-open for this, for example:

with patch("builtins.open", smart_open.open):
    stream = trading.streaming.create_historical_generator_stream(file_path='test.bz2')
    ..
gos-qs commented 2 years ago

Ok thanks, that package looks very useful for loads of my other stuff, is it worth adding a mention to the docs or examplestreaminghistorical.py ?

liampauling commented 2 years ago

💯 I will add something now