A modular adapter to support multiple protocols

superstreamlabs / memphis

Memphis.dev is a highly scalable and effortless data streaming platform

https://docs.memphis.dev

Other

3.26k stars 218 forks source link

A modular adapter to support multiple protocols #849

Open yanivbh1 opened 1 year ago

yanivbh1 commented 1 year ago

Information

Currently, The main protocols supported are NATS using Memphis client SDKs and REST through the REST GW. NATS will continue to be the main protocol, but the REST Gw component should be refactored into a modular adapter where other developers and contributors would be able to add more protocols to that adapter/component.

High-level plan

Protocols roadmap (1-highest)

syslog
gRPC
WS (Issue #479 )
Kafka API
AMQP

GitHub discussion #857

danielsinai commented 1 year ago

Just an idea maybe its possible to implement a plug-able protocol adapter and let the community build any kind of adapter they would like, Pulsar did this pretty well and already have support for kafka, amqp(rabbit, mqtt and more

https://github.com/apache/pulsar/wiki/PIP-41%3A-Pluggable-Protocol-Handler

yanivbh1 commented 1 year ago

Hey @danielsinai, Adding @idanasulinmemphis to this thread as well. Definitely. Most of the pieces in Memphis would be modular and extensible by the community as much as possible, perhaps that's the place to start. We'll create a discussion on that, would love to learn more from your perspective.

GitHub discussion #857

idanasulin2706 commented 1 year ago

Let's continue with the discussion over here https://github.com/orgs/memphisdev/discussions/857 please

g41797 commented 1 year ago

Hi @yanivbh1 I'd like to participate Please assign it to me

g41797 commented 1 year ago

Some notes regarding modular adapter:

Satellite of core server
Process in production, "library" for the testing
Minimal knowledge in memphis "business", just thin host for building blocks
Possible different hosts for environment (stand alone, docker , ...)
Every protocol adapter consists of 1-N blocks (e.g. syslog adapter consists of receiver and publisher)
Dispatcher of adapters also implemented as block(s)
Preferable way of communication between blocks - async messages
Common functionality (helpers/utility "classses") - adapter "infra" will be part of memphis.go
Configurable - (blocks, adapters, conf. for adapter etc) e.g. one host for REST adapter and second for syslog adapter.

Personally I think that using dedicated process per adapter it's preferred mode at least for protocols required web server.

g41797 commented 1 year ago

[Performance] Possible changes in the future on the server side - support unix domain sockets for negotiation with modular adapter

g41797 commented 1 year ago

Dev environment:

memphis-syslog-adapter - similar to rest-gw almost empty - for the fork
fork of memphis.go - additional folder/package adapter

If will be decided that one process should be host for different adapters - also repository

yanivbh1 commented 1 year ago

@g41797 https://github.com/memphisdev/memphis-protocol-adapter I completely agree with your approach, and regarding the name itself, worst-case scenario, it will be changed to your suggested name.

bygui86 commented 1 year ago

Adding my 2 cents here, I think that Kafka API should be the highest priority... currently tons of companies use Kafka, or better to say misuse Kafka and they could replace it with Memphis seamlessly if the Kafka protocol would be supported

yanivbh1 commented 1 year ago

@g41797, the next step of the syslog adapter would be to combine it with the REST gateway, meaning that both will be served by one component to avoid deploying yet another container just for a protocol. It means taking the work that has already done with the REST GW and combining it based on the pattern you implemented in the syslog adapter or vice-versa.

g41797 commented 1 year ago

memphis-protocol-adapter repo is in the same state as memphis cli i don't feel than anyone needs this project also i asked to provide me permissions to merge forked repo with syslog and prepare memphis-protocol-adapter to production but sill didn't get any response i don't see technical restriction to combine rest and syslog, but at least syslog should be tested alone in production mode any requirement from customers side - how this information should be consumed, protocols, number of simm. clients etc i did not see unit tests within rest-gateway. I hope there are external tests. How to combine and/or create simm. tests for two adapters within one process How to validate results Let's talk

Opened #1402

g41797 commented 1 year ago

Issued PR#59 and PR#8

@yanivbh1 & @idanasulinmemphis - your turn