~WIP building out a "flows" layer for incremental update and multi-actor graphics rendering.~
Actually made huge progress on this branch with latency reduction and correctness fixes 🏄🏼
Please don't feel like you have to review all of this code (though that would be super chido)
TL;DR new (performance related) features:
Add a new ui._flows module which begins the removal of data processing / analysis methods from the
chart widget and instead moves them to our new Flow API (in the new module introduced here) and delegating the old chart methods to the respective internal flow.
Most importantly we no longer store the "last read" of an array from shm in an internal chart table (was ._arrays) and instead the ShmArray instance is passed as input and stored in the Flow instance. This greatly simplifies lookup logic such that the display loop now doesn't have to worry about reading shm, it can be done by internal graphics logic as desired. Generally speaking, all previous ._arrays/._graphics lookups are now delegated to the entries in the chart's ._flows table.
incrementally update both step and "flattened" ohlc (to line) curves in local memory (using a new Renderer type) such that shm src data doesn't have to be re-processed on every graphics update cycle
fix the m4 implementation to be uppx (units-per-pixel) driven (thus making it correctly implemented according to the original paper, 🤫) instead of trying to read the px width count in view; it's more sensible (and simple) to just assume the m4 "bin size" by just using the uppx directly instead of trying to effectively compute that value manually and feed it to the algo.. (thus doing the same thing twice..)
NB implementing the incremental updates actually made this change possible since before we weren't downsampling as often nor as "many" datums (at lower uppxs) because the recomputes when not incr updating were too slow. Now we don't have that problem so having less datums and more frequent downsample ops is wayy wayy faster 😎
also let's up drop the now unneeded "log scaling" on px width since granularity concerns are fixed thanks to being able to downsample more often.
adds a new do_append: bool to the display loop which avoids drawing new pixels at high zoom (out) levels until they would be actually noticeable see logic here
TODO deps sync:
[x] we have to sync merging this branch with our forked pyqtgraph branch graphics_pin will will need to be fast forwared on our pike_pin branch.
TODO Factoring:
[x] factor out backend commits that relate to compat with #93 (mostly kraken and ib tweaks iirc)
[x] potentially factor the current mess that is Flow.update_graphics() (😂) into Renderer types and better prepared for sub-actor based path rendering
TODO Polish/UX:
[x] (in ab135d2) ideally we can do the GraphicsObject.draw_last() calls from inside the display loop such that iff the last datum(s) which occupy the final px column worth of y-range will always show the max and min value in that px column's range.
in other words the user can always see the "final datum's" highest and lowest value on screen even if in the range of only occupying one pixel of width
[x] (deferred to #323) there's still an ongoing issue where somehow when we load history that would normally show as a "flat" horizontal line (as segmented flat bars) is pulled from history as duplicate ohlc bars instead (from whatever the last non-flat bar was) and drawn as that duplicate bar shape; this is wrong and results in some seriously ugly history especially on hft freqs. we probably should make a follow up bug for this..
TODO Performance (likely to be deferred to new PR):
[x] (deferred to #324) try out using m4 as the "tracer" for ohlc directly since i'm pretty sure it should return very similar results to our current approach even with a w (bin size) of 0.
[x] (deferred to #325) possibly toy with an alt mxmn sorting approach (such as https://github.com/lemire/pythonmaxmin/blob/master/maxmin.py) at least for the display loop updates such that we aren't resorting on every 1s update which is causing latency right now when viewing 1M + datums..
~WIP building out a "flows" layer for incremental update and multi-actor graphics rendering.~
Actually made huge progress on this branch with latency reduction and correctness fixes 🏄🏼
Please don't feel like you have to review all of this code (though that would be super chido)
TL;DR new (performance related) features:
ui._flowsmodule which begins the removal of data processing / analysis methods from the chart widget and instead moves them to our newFlowAPI (in the new module introduced here) and delegating the old chart methods to the respective internal flow.._arrays) and instead theShmArrayinstance is passed as input and stored in theFlowinstance. This greatly simplifies lookup logic such that the display loop now doesn't have to worry about reading shm, it can be done by internal graphics logic as desired. Generally speaking, all previous._arrays/._graphicslookups are now delegated to the entries in the chart's._flowstable.Renderertype) such that shm src data doesn't have to be re-processed on every graphics update cyclem4implementation to be uppx (units-per-pixel) driven (thus making it correctly implemented according to the original paper, 🤫) instead of trying to read the px width count in view; it's more sensible (and simple) to just assume the m4 "bin size" by just using the uppx directly instead of trying to effectively compute that value manually and feed it to the algo.. (thus doing the same thing twice..)do_append: boolto the display loop which avoids drawing new pixels at high zoom (out) levels until they would be actually noticeable see logic hereTODO deps sync:
pyqtgraphbranchgraphics_pinwill will need to be fast forwared on ourpike_pinbranch.TODO Factoring:
krakenandibtweaks iirc)Flow.update_graphics()(😂) intoRenderertypes and better prepared for sub-actor based path renderingTODO Polish/UX:
GraphicsObject.draw_last()calls from inside the display loop such that iff the last datum(s) which occupy the final px column worth of y-range will always show the max and min value in that px column's range.TODO Performance (likely to be deferred to new PR):
w(bin size) of0.