Cross-actor, shm, (tick) ring buffer

goodboy commented 4 years ago

As part of the fsp subsystem design we're likely going to want to implement some shared memory systems for low latency multi-time frame calcs. I've started a couple design ideas about this in #98 but I do think for sure part of this system will require a fast ring buffer for numpy data:

Here's a starter list of projects to try after some very brief searching

committed to long ago

2016: ringbuffer it's multi-reader/writer, multi-proc aware and has an example with numpy
2015: pyringbuf a c-ext implementing a character buffer

committed recently

2020: pyring pure python impl and has a multi-proc c-types example
2020: ringbuf looks to be built for real-time audio but is single producer/consumer (which is actually fine for our use case afaik) but is built as a cython wrapper around boost (ala C++) libs; numpy example looks good
2021: redo of ringbuf in rust targeting some missing features with ctypes wrapping/integration for python
2020: circular_buffer_numpy seems kinda like a simple wrapper around an array with "pointer" index references kinda like what i've rolled many times before.

futher resources
a scipy mailing list discussion from 09'
comparison of implementing a circular buffer using 3 different data structures
a random walk plot using none other then the great pyqtgraph and a numpy using np.roll().

It may just end up we take a lang from #106 and implement one or try out some designs from above on the new SharedMemory type which has an example using a numpy array. An example wrapper for an older version of this from the scipy cookbooks is here.

disruptor style

The "lurkers" said LMAX already did this best with their disruptor project. I think we can make a very cool variant of this with actors, numpy and numba:

slickin paper
pump vid
there's more resources on the disruptor link :surfer:
data_pipe is another one to check out (though not sure it'll have numpy support at all).
- Looks to imply it has some disruptor style examples.

As always, lurkers please chime.

goodboy commented 4 years ago

The "lurkers" said LMAX already did this best with their disruptor project:

slickin paper
pump vid
there's more resources on the disruptor link :surfer:

I think we can make a very cool variant of this with actors, numpy and numba.

goodboy commented 3 years ago

data_pipe is another one to check out (though not sure it'll have numpy support at all).

Looks to imply it has some disruptor style examples.

goodboy commented 3 years ago

This definitely can tie in with #192 @guilledk

pikers / piker