`marketstore` OHCLV storage and client integration

guilledk commented 2 years ago

Update outdated ingest backend and replace the in-repo async wrapped marketstore client with the new: https://github.com/pikers/anyio-marketstore

Includes support for a tractor based supervisor actor and thus resolves #143 which was merged in via the sub-PR #284.

[x] re-add destroy cli piker cmd (now implemented as Storage.delete_ts())
[ ] tick ingest to marketstore from brokerd feeds:
- [ ] experiment with techtonicdb schema and if it can be used with the aggregator plugin (got a feeling we'll need to write at least a 1Sec bucket in order for this to work looking at the code:
- aggregate() calls into model.FromTrades() and FromTrades() only accepts a min step of 1Sec
- we might be able to either write a golang plugin/extension to do it from pure tick data or we can make this part of a piker actor?
[x] ~ohlcv ingest and load from multiple backends:~ this was added as part of the new history in #308.
- ib seems like the natural place to start since 1s is already offered in their history
- we'll need separate series for dark vs. regular vlm
- [ ] we'll likely want to write a diffing and syncing system to validate new captured histories from real-time runs (which almost always are slightly different)
- kraken has tick history which has been a todo for a while
[ ] benchmark binance tick feed against binancefeeder

[ ] probably deliver back net-socket info over the ctx.started() call?
- [ ] grprc socket
- [ ] ws addr
[x] cli support for running pikerd with the tsdb stuff spawned?
- [x] maybe a --tsdb or --data or something?
[ ] figure out what to do with the mkts.yaml config?
- we could push a template from code to the user dir?
- or should we just always gen it from python?
- step by step interaction test lisit:
- [x] pikerd --tsdb should raise DockerNotStarted and appropriate perms error on no sudo (for now)
- [x] ctrl-c should kill container instance
  - [ ] this does not really work when run with sudo since we relinquish root perms in order to create shms..

goodboy commented 2 years ago

@guilledk can you rebase this amigo?

goodboy commented 2 years ago

Merged in the super stuff from #284 and will begin 1s OHLC ingest from ib shortly 🏄🏼

goodboy commented 2 years ago

Oh right and I have to go through the todos and update!

Mostly just want get a first set of eyes on the code.

goodboy commented 2 years ago

Replaced by #308 🏄🏼

pikers / piker