superstreamlabs / memphis

Memphis.dev is a highly scalable and effortless data streaming platform
https://docs.memphis.dev
Other
3.25k stars 216 forks source link

MongoDB Removal #625

Closed yanivbh1 closed 1 year ago

yanivbh1 commented 1 year ago

Motivation

Remove any major dependency for running in air-gapped environments, smaller footprint, and fewer points of failure.

Key points to consider

DrBlury commented 1 year ago

I just stumbled upon this repository, so forgive me any unknowingness. But isn't the MongoDB just used for UI? What if the UI is a separate container with a MongoDB next to it?

Maybe I don't want/need the UI but only the API that the UI uses so I can integrate it into my already existing monitoring stack.

So instead of removing MongoDB altogether and potentially moving to a in-service database or something like that... Try to think about those use-cases! Flexibility is nice. Having control over your database is nice too.

I worked with Apache Kafka extensively in the last few years with multiple clients so I know a thing or two on how people use or misuse it. :D If I can help somehow with brainstorming or actual development etc. please let me know.

Either via Telegram: @DrBlury or Discord: DrBlury#6715

yanivbh1 commented 1 year ago

Hey @DrBlury, great to have you here! Would be great to have you in our discord as well. With your experience around Kafka, it would be great to learn more feedback and how we can improve. You are absolutely right in your understanding of the purpose of the DB, and that is exactly what we thought about. The ultimate solution would be hybrid - you can use either your own external Mongo, for example, or the in-service one in Memphis. A good point that should be considered is if you don't want GUI at all. I am adding those points to the description. Thank you.

yanivbh1 commented 1 year ago

Released in v1.0.0 @DrBlury FYI