Paper trade improvements

algorandpa commented 1 year ago

Added ledger and positioning support for paper mode
Ensure we immediately write to ledger and update position in real time
Added local txn state in Paperboi

Test suite todo:

[x] clear a paper order, ensure the pp shows up in pps.toml, stop and start pikerd, ensure the pp is relayed on open_ems() connect
[ ] ensure after a return to a net zero position, the pps.toml does not get a leaked account entry as per comment https://github.com/pikers/piker/pull/462#issuecomment-1438852266
[x] when loading a paper position after a full restart of the service stack (i.e. shutting down and restarting pikerd) if you make a new clear to begin "closing out" (i.e. reducing the dst asset) the paper engine is not sending the correct BrokerdPosition state as per the comment https://github.com/pikers/piker/pull/462#discussion_r1113400683, as such we need to add test code to process the trade_stream: tractor.MsgStream msgs and ensure the eventual last position msg matches the state in the pps.toml

Future dark-order and alert storage todo test suite (maybe to do in a follow up PR):

~submit an order that does not clear, stop and start pikerd, ensure order is re-registered in emsd~ => deferred to #463 ~will likely requires Order serialization to file system and load~ => deferred to #464

goodboy commented 1 year ago

@algorandpa while i'm testing and reviewing this you might as well make your follow issue(s) for that last bullet and whatever else 🏄🏼

algorandpa commented 1 year ago

Issue created for second task here @goodboy

goodboy commented 1 year ago

Oh and for the trailing white space issue you seem to have; try out this vim config snippet:

set list listchars=tab:\ \ ,extends:$,precedes:^,trail:-

goodboy commented 1 year ago

When this lands I believe we'll be completing the bullet from the paper engine support section in https://github.com/pikers/piker/issues/345 🥳

At the time when we merge this PR you can check off that bullet and put a sub-bullet linking to this I'd say.

goodboy commented 1 year ago

Ok as per discussion on matrix, I found a couple issues:

[x] any closed paper position will leak a TOML section entry in pps.toml

screenshot-2023-02-21_12-24-43

[x] when loading a paper position after a full restart of the service stack (i.e. shutting down and restarting pikerd) if you make a new clear to begin "closing out" (i.e. reducing the dst asset) ~the order mode UI and ppu line act as though no previous position was loaded~ actually it's a bug in the paper engine, see comments below in the code, even though the ledger file and pps.toml are FOR SURE correct.

~this is likely some kind of state bug inside piker.ui.order_mode we need to figure out; unfortunately I don't think we can easily test this without also loading the Qt runtime..~ nope, it's a position msging bug inside the paper engine .fake_fill() method. maybe it's getting close to time to start leveraging pytest-qt 🏄🏼

seems like the issue is messaging from the paperboi to the emsd (i think):


Feb 21 12:40:17 (emsd[3acd02], 6021, _emsd_main)) [INFO] piker.clearing._ems _ems.py:1044 Received order cmd:
{'account': 'paper',
'action': 'buy',
'brokers': ['binance'],
'exec_mode': 'live',
'oid': 'a45a7742-108b-46e5-82b8-9689ca474b4e',
'price': 24455.41,
'size': 0.04093,
'symbol': 'btcusdt.binance'}
Feb 21 12:40:17 (emsd[3acd02], 6021, _emsd_main)) [INFO] piker.clearing._ems _ems.py:1206 Sending live order to binance:
BrokerdOrder(oid='a45a7742-108b-46e5-82b8-9689ca474b4e', account='paper', time_ns=1677001217069566678, symbol='btcusdt', price=24455.41, size=0.04093, action='buy', reqid=None)
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:694 Received broker trade event:
{'account': '',
'name': 'ack',
'oid': 'a45a7742-108b-46e5-82b8-9689ca474b4e',
'reqid': '8cec6566-4de0-4d3a-9a27-35aae5f515fe'}
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:739 Rx ACK for order
oid: a45a7742-108b-46e5-82b8-9689ca474b4e -> reqid: 8cec6566-4de0-4d3a-9a27-35aae5f515fe
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:694 Received broker trade event:
{'account': 'paper',
'broker_details': {'name': 'paperboi'},
'filled': 0.0,
'name': 'status',
'reason': 'paper_trigger',
'remaining': 0.04093,
'reqid': '8cec6566-4de0-4d3a-9a27-35aae5f515fe',
'status': 'open',
'time_ns': 1677001217121686778}
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:876 binance opened order BrokerdStatus(reqid='8cec6566-4de0-4d3a-9a27-35aae5f515fe', time_ns=1677001217121686778, status='open', account='paper', name='status', filled=0.0, reason='paper_trigger', remaining=0.04093, broker_details={'name': 'paperboi'})
Feb 21 12:40:17 (paperboi.binance[be4a8e], 6026, piker.clearing._paper_engine.handle_order_requests)) [INFO] piker.clearing._paper_engine _paper_engine.py:238 Fake filling order:
BrokerdFill(broker_time=1677001217.173384, reqid='8cec6566-4de0-4d3a-9a27-35aae5f515fe', time_ns=1677001217173376271, size=0.04093, price=24450.87, name='fill', action='buy', broker_details={'paper_info': {'oid': 'a45a7742-108b-46e5-82b8-9689ca474b4e'}, 'name': 'binance_paper'})
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:694 Received broker trade event:
{'action': 'buy',
'broker_details': {'name': 'binance_paper',
            'paper_info': {'oid': 'a45a7742-108b-46e5-82b8-9689ca474b4e'}},
'broker_time': 1677001217.173384,
'name': 'fill',
'price': 24450.87,
'reqid': '8cec6566-4de0-4d3a-9a27-35aae5f515fe',
'size': 0.04093,
'time_ns': 1677001217173376271}
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:895 Fill for a45a7742-108b-46e5-82b8-9689ca474b4e cleared with:
{'action': 'buy',
'broker_details': {'name': 'binance_paper',
            'paper_info': {'oid': 'a45a7742-108b-46e5-82b8-9689ca474b4e'}},
'broker_time': 1677001217.173384,
'name': 'fill',
'price': 24450.87,
'reqid': '8cec6566-4de0-4d3a-9a27-35aae5f515fe',
'size': 0.04093,
'time_ns': 1677001217173376271}
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:694 Received broker trade event:
{'account': 'paper',
'broker_details': {'name': ''},
'filled': 0.04093,
'name': 'status',
'reason': '',
'remaining': 0,
'reqid': '8cec6566-4de0-4d3a-9a27-35aae5f515fe',
'status': 'closed',
'time_ns': 1677001217174046682}
Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:866 Execution for a45a7742-108b-46e5-82b8-9689ca474b4e is complete!
Ledger load took 0.0017094612121582031s
Updating ledger for /home/goodboy/.config/piker/ledgers/trades_binance_paper.toml:

Updating pps.toml for <module 'posixpath' from '/usr/lib/python3.10/posixpath.py'>:

Feb 21 12:40:17 (paperboi.binance[be4a8e], 6026, piker.clearing._paper_engine.handle_order_requests)) [INFO] piker.clearing._paper_engine _paper_engine.py:507 Submitted paper LIMIT 8cec6566-4de0-4d3a-9a27-35aae5f515fe: BrokerdOrder(oid='a45a7742-108b-46e5-82b8-9689ca474b4e', account='paper', time_ns=1677001217069566678, symbol='btcusdt', price=24455.41, size=0.04093, action='buy', reqid=None) Feb 21 12:40:17 (emsd[3acd02], 6021, piker.clearing._ems.translate_and_relay_brokerd_events)) [INFO] piker.clearing._ems _ems.py:694 Received broker trade event: {'account': 'paper', 'avg_price': 24450.87, 'broker': 'binance', 'currency': '', 'name': 'position', 'size': 0.04093, 'symbol': 'btcusdt.binance'}

goodboy commented 1 year ago

Lol, sorry hit the wrong button on that last comment 😂

Ok I found the issue for the second bullet above I'm going to comment on the code backlinking to that comment.

goodboy commented 1 year ago

Tested, and reviewed code, I have some minor comments but I will address them in a PR of my own, decimalization_take_2 branch.

I presume you want to make that vs. master once this lands yah?

goodboy commented 1 year ago

Ok so did some manual testing and everything is working minus what's expected not to, which i'm going to state below and feel free to correct:

[x] still leaking a pps.toml section header **only when we make a position, stop the service tree, boot back up, zero the pp.
- TODO: we should make a follow up issue both to make a test which catches this and obviously a fix for it!
[ ] we have a position size numerics issue to do with using float which @guilledk resolves in #470

Remaining ToDo for this to land:

[x] adjustments to code as per my last review
[x] follow up issue for the 1st bullet above

again excellent work getting this test harness started @algorandpa

goodboy commented 1 year ago

Huh, so interesting..

I ref to #470 i am seeing that the rounding/precision issues only seem to be in the .pp.PpTable stuff since the order mode UI does indeed wipe on zeroing after closing out the pos over multiple pikerd sessions.

screenshot-2023-02-28_13-40-02

goodboy commented 1 year ago

Though i'm sure @guilledk is totally aware of the details, just putting this for myself and the knowledge of others as per the fix landing in #470.

Turns out the msging layer seems to be completely fine in terms of zeroing pps despite using floats throughout sizing arithmetic; the issue seems to be purely in our .pp.PpTable size storage and arithmetic sub-sys.

As an example here i was able to get the precision error to linger in the pps.toml but in the UI and at the msging layer there was a zeroed pp every time i started a new session and the accum_size = 2.7755575615628914e-17 was present in the clearing table:

[binance.paper."btcusdt.binance"]
size = 2.7755575615628914e-17
ppu = 23489.324999999997
bsuid = "btcusdt"
clears = [
 { dt = "2023-02-28T18:36:42.360930+00:00", ppu = 23455.5, accum_size = -0.04264, price = 23455.5, size = -0.04264, cost = 0, tid = "34323ca0-277b-4f9f-b77e-1ef4718c49c5" },
 { dt = "2023-02-28T18:36:44.500479+00:00", ppu = 23456.435, accum_size = -0.08528, price = 23457.37, size = -0.04264, cost = 0, tid = "9996508c-b94b-489f-910f-c9360c9ce031" },
 { dt = "2023-02-28T18:36:46.400183+00:00", ppu = 23456.00666666667, accum_size = -0.12791999999999998, price = 23455.15, size = -0.04264, cost = 0, tid = "15dfe435-d54b-450d-b1ec-21db6e535c17" },
 { dt = "2023-02-28T18:38:37.989248+00:00", ppu = 23462.75, accum_size = 2.7755575615628914e-17, price = 23462.75, size = 0.12792, cost = 0, tid = "138dcf2b-5ab9-43b3-bb45-0465da730ac9" },
 { dt = "2023-02-28T18:44:02.788944+00:00", ppu = 23480.89, accum_size = 0.04258000000000003, price = 23480.89, size = 0.04258, cost = 0, tid = "e098b4d2-72b2-41a9-b1f4-e74f6c457c56" },
 { dt = "2023-02-28T18:44:11.890799+00:00", ppu = 23480.89, accum_size = 2.7755575615628914e-17, price = 23482.39, size = -0.04258, cost = 0, tid = "672167d1-ee78-4e54-91a6-7fc63f96349f" },
 { dt = "2023-02-28T18:44:41.429072+00:00", ppu = 23491.16, accum_size = 0.042570000000000024, price = 23491.16, size = 0.04257, cost = 0, tid = "df3eeead-e6ad-477a-9720-c5143ace03cf" },
 { dt = "2023-02-28T18:44:44.812730+00:00", ppu = 23489.324999999997, accum_size = 0.08514000000000002, price = 23487.49, size = 0.04257, cost = 0, tid = "30f471e6-e1c0-484b-91cd-cd9c9aec8804" },
 { dt = "2023-02-28T18:44:48.019707+00:00", ppu = 23489.324999999997, accum_size = 0.042570000000000024, price = 23488.71, size = -0.04257, cost = 0, tid = "6f6f2a34-3fe0-436b-9896-4874172cbfc2" },
 { dt = "2023-02-28T18:45:31.141610+00:00", ppu = 23489.324999999997, accum_size = 2.7755575615628914e-17, price = 23480.55, size = -0.04257, cost = 0, tid = "46440830-caed-4b76-b6b3-0ac27fa7b054" },
]

pikers / piker