We're on the cusp of introducing real-time charting and with that the ability to easily enable forward testing and sophisticated types of back testing (such as WFO). This is possible with the right broker but there's some design decisions to be made about how data feeds are managed for IPC and minimum latency.
There are 3 main data sinks I can envision from the outset:
There's a lot to research and test wrt to interchange formats (goodboy/tractor#58), IPC protocol details, and frankly a lot of the stuff apache arrow and flight are built to address. I wanted to start describing some feed architectures and conventional problems that will be of note.
Shared memory updates
One the designs we should experiment with heavily is shared mem IPC for numpy array passing since that is likely to stay (for now) our primary data structure format due to its wide adoption in the data community. In particular using an architecture where near term data is written to a buffer directly by the broker feed process such that latency can be minimized for reads from other local processes (eg. to update graphics UIs and local downstream processing pipelines asap) versus there being larger delays for processes with lesser latency constraints (eg. downstream feeds used for monitoring or shared with a human trader who's reaction time is much slower).
it would be super interesting to develop a trio integration with the object store system (which has timeouts and stuff) but I'm not sure if you can unseal and overwrite old buffers (or if that's even the right way to use it) to implement a ring.
benchmarks of shared mem versus just using standard IPC interchange (like msgpack-numpy) over some zero-copy transport (like pipes on linux?) to compare latency.
We're on the cusp of introducing real-time charting and with that the ability to easily enable forward testing and sophisticated types of back testing (such as WFO). This is possible with the right broker but there's some design decisions to be made about how data feeds are managed for IPC and minimum latency.
There are 3 main data sinks I can envision from the outset:
marketstore
,influxdb
, SQL stores,techtonicdb
)tractor
actors runningnumpy
computations, ML framework endpoints)There's a lot to research and test wrt to interchange formats (goodboy/tractor#58), IPC protocol details, and frankly a lot of the stuff apache
arrow
andflight
are built to address. I wanted to start describing some feed architectures and conventional problems that will be of note.Shared memory updates
One the designs we should experiment with heavily is shared mem IPC for
numpy
array passing since that is likely to stay (for now) our primary data structure format due to its wide adoption in the data community. In particular using an architecture where near term data is written to a buffer directly by the broker feed process such that latency can be minimized for reads from other local processes (eg. to update graphics UIs and local downstream processing pipelines asap) versus there being larger delays for processes with lesser latency constraints (eg. downstream feeds used for monitoring or shared with a human trader who's reaction time is much slower).Some notes on all this:
considerations for the problem of false sharing
disruptor
is an example of a production grade solution in java developed by the LMAX exchange.this system could of course be rolled by hand using the std lib's interface that works well with
numpy
arrays (which I've actually deployed before but with a ring buffer interface)pyarrow
already has a zero-copy api for this called plasma (also note implications forpd.DataFrame
zero copy support).trio
integration with the object store system (which has timeouts and stuff) but I'm not sure if you can unseal and overwrite old buffers (or if that's even the right way to use it) to implement a ring.benchmarks of shared mem versus just using standard IPC interchange (like msgpack-numpy) over some zero-copy transport (like pipes on linux?) to compare latency.