search

alpacahq / marketstore

DataFrame Server for Financial Timeseries Data
Apache License 2.0
1.89k stars 232 forks source link

Feature request #132

Open Swoorup opened 6 years ago

Swoorup commented 6 years ago

Does this support inserting historical order book data?

bolobox commented 5 years ago

+1 for order book storage support in marketstore!

umitanuki commented 5 years ago

Yes that's what we want, too! The data format is going be more complicated so we need some design time.

athenawisdoms commented 4 years ago

This is an excellent idea!

Any updates on order book data integration, please?

If not, any recommendations on how to currently best store historical order book data?

@umitanuki

goodboy commented 4 years ago

@athenawisdoms you might want to give techtonicDB a try for an out of the box solution with an excellent storage format.

However, I am already story tick data using the <symbol>/1Sec/TICK time bucket (note the Nanoseconds subtleties from #325) and it is working quite well. Adding an additional Level or Price field for each row entry should give you something similar to what techtonic provides though of course with a much less compact format.

athenawisdoms commented 4 years ago

@goodboy Great recommendation, will be sure to try it out.

If we want to store both trades and order book data, will you suggest using marketstore for trades and techtonicDB for order book? Or sticking to just one (i.e. marketstore) like you have done?

Additionally. have you looked into the data compression ratio for order book data when using marketstore the way you are currently storing, compared to a more widely used time series (eg. timescaledb, influxdb)?

Thanks!

goodboy commented 4 years ago

@athenawisdoms I actually am using both at the moment (techtonicDB as well just don't have code up yet). I'm currently delegating to underlying broker systems to store trade history so that hasn't been a requirement yet.

have you looked into the data compression ratio for order book data when using marketstore the way you are currently storing, compared to a more widely used time series (eg. timescaledb, influxdb)?

I haven't in detail but the marketstore design docs might provide insight. I have done some reading on both timescaledb and influxdb. You can see a small list of options we've been auditing here: https://github.com/pikers/piker/issues/90

goodboy commented 4 years ago

Pretty sure this pertains to the new tick to bar conversion that's just been added in #399.

Tick is usually the level 1 of the book (aka the bid/ask spread) so if tick aggregation has now been added then we have a very basic top of book time series going on.