Data Ingest: Reading

[echavemann]

Prereq: #45 design - need to have the compression system figured out.

This ticket is to create reader objects that we use to invert the dependency of reading data. These should be either Trade or Snapshot reader (probably need a base class), and then be able to read each line, read entire files, and be able to have a .next structure so it can be run forward continuously until we run out of data. Id get reviews on the API first, and then get ticks on the implementations. Please attach a performance benchmark (you can use hyperfine) to the PR.

Integration tests please.

[rewong03]

imo i think this ticket boils down to writing Trade/SnapshotMessage Reader/Writer classes

• Data should be compressed/decompressed using pyzstd (https://pyzstd.readthedocs.io/en/stable/)
• Writer should ideally store data in a binary format (compressed)
• Reader should ideally have a generator method to read messages from a file without loading the whole file in memory. it ideally also should have a method to do this over a list of files
• Here's some code for serializing snapshot, here's some code for deserializing snapshots from a stream, here's code for serializing trades, here's code for deserializing trades

[rewong03]

also existing code and tests will need to be refactored :)

[Jiarui-Cao]

I see that the trades data client already has a get_trades_from_file function, and the updates data client already has the stream_updates function. Can I just add stream_trades to HistoricalTradesDataClient, and get_updates_from_file to HistoricalUpdatesDataClient?

Then I can have tick-by-tick trades and updates loader built on top of these two data clients to feed to target loader #49