Rekt pps? problem? => `piker.accounting`

[goodboy]

Even moar fixes after #470, #475, #484 which mega broke the old pps.toml management for live backend 💥 😶‍🌫️ 🤕

Turns out this might be the reason to start assaulting the revamp of our accounting tooling as per our now long list of desirables: https://github.com/pikers/piker/labels/accounting

Testing enhancements (some tasks deferred)

• [x] rework test set to use pytest.mark.parametrize so we can more easily use common harness logic for darks and alert testing.

• [x] reorg the ahab API docker supervisor endpoints to be decoupled from open_pikerd() and able to accept arbitrary config overrides for both testing and distributed setups.

• [ ] dark order and alert clearing

  • 466 and what we're planning to do for offline storage of level

    lines

• [ ] add a basic accounting of position only test set?

  • deferred to #512 testing section

tasks and bugs getting deferred..

• [ ] pre-existing live order loading is borked on first chart edit from order mode: https://github.com/pikers/piker/issues/502

• [ ] final removal of Symbol and .symbol semantics -> summarized in #517

  • [ ] drop from clearing msg set.
  • [ ] order_mode .symbol final usage..?

• [ ] make a follow up issue to add a Flume.Cascade type for fsps which describes stream IO between flumes:

• [ ] deribit still need full get_mkt_info() and FeedInit

• [x] sync up the new data.validate stuff with old brokercheck idea in

  387, #393 -> add an actual test?

dependency changes needed for this to land

• [x] move to new TOML libs (tomlkit for styling and tomli-w for fast writing) B)

  • [x] break the shoddy toml writing hacks we have into a new submod..
  • [x] actually remove the toml dep and go with tomli, tomli-w and tomlkit B)
  • [x] move to using tomlkit which does support writing inline tables (as well as lotsa other great stuff) as requested in #496 B)
  • [x] For now pin to our fork! (https://github.com/pikers/tomlkit/tree/piker_pin)
    • need to land all local changes and pin the requirement!
  • [x] make PR to upstream -> https://github.com/sdispater/tomlkit/pull/294

• [ ] https://github.com/goodboy/tractor/pull/357

  • [x] at least move the piker pin to this history to get CI green

---

Unrelated to .accounting changes

• complete rework of .data._web_bs implementation to avoid the use of backend code using async generators within async context managers.
  • see 59743b7b7 and follow up commits
  • yeah probably should have broken this out into a separate PR but too many devs are too busy to review it rn so there's not a lot of point :joy:

---

NEW SUBSYSTEM: the piker.accounting (subpkg) :surfer:

• [x] move all .data._source and piker.pp mod contents into a much better organized subsystem API -> piker.accounting :surfer:

  • ._mktinfo contains MktPair and Asset primitive as well as new .fqme acronym (fully qualified market endpoint) processing.
  • ._pos contains Position and PpTable loading and processing
  • ._ledger adds TransactionLedger and Transaction loading and sync.
  • .cli adds a ledger <broker>.<accountname> syncing prototype cli which can be used to dump position summaries to console.

• [x] try to see how many bullets we can nail from https://github.com/pikers/piker/issues/345

  • [x] offer .write_config() to (the new) TransactionLedger and PpTable for immediate writes.

  • further follow ups: #510, #511, #513, #515

  • [x] drop the singleton pps.toml instead breaking into multiple account files per backend,

  • [x] per broker-account position tracking via separate toml files named as $HOME/.config/piker/accounting/account.<brokername>.<(sub)accountname>.toml

    • eg. account.binance.paper.toml, account.kraken.spot.toml which are named from the accounts listed in a user's brokers.toml B)

  • [x] adjust internal account-position files to not include account name in table schema: i.e. no longer any kraken.paper in any section keys.

• [x] Prototype ledger cli using typer with follow up to come with #512

  • [x] add typer and rich as deps!

• [x] Move our data feed layer to new MktPair market info struct as well as support a new "backend validation system" using the new .data.validate.FeedInit

  • we now expect stream_quotes() to deliver a pedantic list[FeedInit] on startup instead of a random dict of some schema :surfer:

  • all backends should provide a get_mkt_info() endpoint now which does the work of constructing a MktPair from an input search pattern (str) and is also expected to construct the MktPair.src / .dst fields which in the ideal case are .accounting._mktinfo.Asset refs.

  • [x] move all internal usage of the old Symbol type over to this as well as rename as much as we can of fqsn to the new fqme (fully qualified market endpoint)

  • [x] data layer including Flume.symbol -> .mkt: MktPair

  • [x] clearing engine

  • [x] fsp engine

  • [x] change Position.symbol -> .mkt

  • [x] UI layers: _chart, _display _fsp.. etc.

  • [x] Drop ui._chart.LinkedSplits.symbol outright and finally!

  • [x] port all backends as much as is possible to this new interface

  • [x] drop "good job" msg from .data.validate since it should be default now B)

  • [x] kraken, ib for true full live support

  • [x] binance, kucoin for read-onlys implementation, but i don't have a live feed..

  • [x] make an issue for questrade? -> #509

  • [x] bring in backend checks from #393 and add to a test set for superficial audit of all broker/data providers.

  • ended up just tossing in warning log msgs to data.validate stuff and not sure we really need explicit test set?

---

Outstanding/Related issues/bugs that should be resolved here:

• [x] https://github.com/pikers/piker/issues/373 closed for #515

• [x] https://github.com/pikers/piker/issues/472 fixed here and prior

• [x] https://github.com/pikers/piker/issues/496

  • linked in our PR and issues for follow up B)

• [x] ib forex is currently borked since the move to MktPair (b44b0915)

  • [x] fix no-vlm relay as well so data layer / sampling / charts behave as before!
  • it's actually fine graphically as is (an empty vlm plot) but it's burning a shm allocation...
  • [x] fix cmdty and other non-vlm contract live feed startup.. bad con type matching.

• [x] kucoin borked on search result feed selection-load. (ff74d47f)

---

discovered BUGs tofix, issues likely to re-issued and defer, new feature ideas:

• [x] kraken outstanding stuff from #263 -> moved to #515

---

extra follow ToDos (birds with stones):

• follow up .accounting subsys summary refinements and bug fixes in #515

• EMS layer: unify the messaging layer to a single msg set: #514

• CLI piker ledger (sync) development and nice-to-haves

  • [x] make follow up issue: #512
  • [x] make follow up to move to typer for all CLIs: #513

[goodboy]

Lul, gotta drop that pytest_trio import..

[goodboy]

Ok so as per discussion in dev chat I'm going to cowboy land this massive change set without extensive review bc,

• there aren't enough core devs around to give it the attention rn
• the bug fix creep got too big and would require a lot of commit factoring and sub-PRs to be made (which i don't really have the time or patience to go through rn 😂 ) along the scale of what happened in #420 ..
• there's enough critical backend stuff that was fixed and the new .accounting pkg is important enough to get on the main branch asap.
• there is a test set which has caught pretty much all regressions and all such issues have been resolved.
• everything else that wasn't finalized has been documented in follow up issues.
• i've been using this code in production for weeks and am aware of any oustanding critical live trading problems.

Hopefully this won't happen again 😹