Point72 / csp

csp is a high performance reactive stream processing library, written in C++ and Python
https://github.com/Point72/csp/wiki
Apache License 2.0
150 stars 27 forks source link

Expand websocket adapter, ensure connections/disconnections can be done dynamically, establish better pattern for dynamic subscriptions for use in other adapters #277

Open timkpaine opened 3 weeks ago

timkpaine commented 3 weeks ago

Many adapters have static, wiring-time connection requirements, e.g. you need to provide a url or channel or topics at wiring time and cant make new connections dynamically. Starting with the websocket adapter where it is more (most?) relevant, we should establish a better pattern for dynamic subscriptions. This will be of particular importance in the upcoming redis adapter.

We may want to develop a common structure for subscriptions, though it will like have implementation-dependent details. As a simple example for websockets:

class ConnectionRequest(struct):
    url: string
    action: ActionType # Connect, Disconnect, Ping, etc
    persistent: bool  # e.g. 
    on_connect_payload: Any
   ...

Then adapters will have the following structure:

%%{init: {'flowchart': {'defaultRenderer': 'elk'}}}%%

flowchart LR
    requests["Requests for new connections/paths"]
    data_from["Data from existing connections/paths"]
    data_to["Data to connections/paths"]
    adapter["Adapter/Manager"]
    status["Status or Status of Requests"]
    requests --> adapter
    data_to --> adapter
    adapter --> status
    adapter --> data_from

For redis its even more complicated, but still the same overall pattern

%%{init: {'flowchart': {'defaultRenderer': 'elk'}}}%%

flowchart LR
    key_and_data_to_write["Key + Data to write"]
    key_hash_data_to_write["Key/Hash + Data to write"]
    key_to_read["Key to read"]
    key_and_hash_to_read["Key/Hash to read"]
    key_to_sub["Key to sub"]
    adapter["Adapter/Manager"]
    data["Data from read request"]
    data_from_sub["Data from sub"] 
    status["Status or Status of Request"]
    key_and_data_to_write --> adapter
    key_hash_data_to_write --> adapter
    key_to_read --> adapter
    key_and_hash_to_read --> adapter
    key_to_sub --> adapter
    adapter --> status
    adapter --> data
    adapter --> data_from_sub