First draft justifications and theoretical underpinning
The main focus of this effort is to provide a simple and low(er) level API for the management of shared memory array/buffers for the purposes of real-time processing.
By leveraging the semantics and constraints of structured concurrency, we can provide an API that avoids the presence of data races in the context of concurrent stream and signal processing and the resource management of shared data structures without the need for special additional "shared resource subsystems" such as the stdlib's ResourceTracker or the public APIs for SharedMemoryManager.
In an SC multi-processing system (which tractoris) such auxiliary subsystems are simply unrequired since the lifetime management of processes and their scheduled tasks are already explicitly defined through nursery constructs 🏄🏼
As the most obvious example, a very simple set of premises provided by our SC actors enables a data race free shm management system without the need for any "special" resource tracking:
a parent actor is spawned and allocates a shared memory buffer which it, and it alone is given write access
the parent spawns (multiple) subactor(s) which are given read-only access to the shm
=> due to no child actor being able to outlive its parent, the configuration of processes and the shm buffer already implicitly implements a readers-writer lock; there is zero concern for data races during shm segment management since no child can be left reading after the resource has been unlinked. Further, no special tracking of the shm needs to be conducted as long as the parent/alocator de-allocates the segment on exit.
Outstanding bug-issues to be resolved for this implementation
There is a slew of related piker issues that drive the need to better formalize this API:
Near-term ToDo list for first draft landing as a documented public API:
[ ] first implementation factored from the piker project is in #338 and includes:
[x] default, native support for the stdlib's ShareableList
[ ] necessary adjustments to name numpy support optional
[ ] solving the shm allocation bugs in the first 2 bullets of the bug-issues section above:
[ ] likely we need to write a definitive example/test which demonstrates the buffer leaking / unlinking failures in the stdlib, example was flung in https://github.com/pikers/piker/issues/332
[ ] a first advanced example would be nice to have that demonstrates the readers-write lock style mentioned in the first section here
potentially the example application could be a streaming search or sorting algo:
First draft justifications and theoretical underpinning
The main focus of this effort is to provide a simple and low(er) level API for the management of shared memory array/buffers for the purposes of real-time processing.
By leveraging the semantics and constraints of structured concurrency, we can provide an API that avoids the presence of data races in the context of concurrent stream and signal processing and the resource management of shared data structures without the need for special additional "shared resource subsystems" such as the stdlib's
ResourceTrackeror the public APIs forSharedMemoryManager.In an SC multi-processing system (which
tractoris) such auxiliary subsystems are simply unrequired since the lifetime management of processes and their scheduled tasks are already explicitly defined through nursery constructs 🏄🏼As the most obvious example, a very simple set of premises provided by our SC actors enables a data race free shm management system without the need for any "special" resource tracking:
=> due to no child actor being able to outlive its parent, the configuration of processes and the shm buffer already implicitly implements a readers-writer lock; there is zero concern for data races during shm segment management since no child can be left reading after the resource has been unlinked. Further, no special tracking of the shm needs to be conducted as long as the parent/alocator de-allocates the segment on exit.
Outstanding bug-issues to be resolved for this implementation
There is a slew of related
pikerissues that drive the need to better formalize this API:Motivating feature enhancements
Near-term ToDo list for first draft landing as a documented public API:
[ ] first implementation factored from the
pikerproject is in #338 and includes:ShareableListnumpysupport optional[ ] solving the shm allocation bugs in the first 2 bullets of the bug-issues section above:
[ ] a first advanced example would be nice to have that demonstrates the readers-write lock style mentioned in the first section here