search

INTERSECT-SDK / python-sdk

Interconnected Science Ecosystem - Software Development Kit (INTERSECT-SDK)
https://intersect-python-sdk.readthedocs.io
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

Support Apache Pulsar protocol for SDK broker communiation #6

Open marshallmcdonnell opened 4 months ago

marshallmcdonnell commented 4 months ago

Create examples for pulsar here:

Then, this will pick up the end-2-end test of the example. Add pulsar run_example_test here:

Then implement the broker client for pulsar here:

Lance-Drane commented 4 months ago

Regarding the binary encoding issue, Pulsar is one of the protocols which supports protocol-level headers, so support will be guaranteed to continue once we add it.

Until that issue is tackled, I wouldn't worry too much about the precise implementation details, as the BrokerClient interface is still too narrow (publish() only accepts a raw byte payload, the subscribe() callback functions will still only provide the raw byte payloads, etc.).

Once the general APIs have been defined for other protocols, I don't expect it would be difficult at all to refactor the Pulsar logic to send metadata in the headers.