ezmsg-org / ezmsg

Pure-Python DAG-based high-performance SHM-backed pub-sub and multi-processing pattern
https://ezmsg.readthedocs.io/en/latest/
MIT License
10 stars 5 forks source link

TCP connections could be more efficient. #5

Open griffinmilsap opened 1 year ago

griffinmilsap commented 1 year ago

Given the following scenario:

Currently, all subscribers will receive the TCP transmission and each overwrite the cache with the same (identical) received message that was serialized, sent, received, deserialized and copied multiple times needlessly. Fixing this would involve giving MessageCache a socket/server that it could synchronize with other MessageCaches running on other machines/processes. This greatly complicates and fundamentally changes the backend message passing design, and may actually degrade performance for SHM and local caching significantly.

WORKAROUND/DESIGN TIP: This performance issue can be worked around by creating a re-publisher that receives TCP messages once, then re-publishes them into local process/cache such that other subscribers in this process have direct cache access to the message that was serialized only once from the source process.

griffinmilsap commented 1 year ago

This bug was mostly addressed (for multiple topics in the same unit) by v3.1.2 with message caching. This issue might be fully-addressed by adding Collection Stream republishers as discussed in #11