NATS to Mongodb (vise versa)

ahmed-adly-khalil commented 1 year ago

Hello, This is really neat work, did you think about the other way around? means syncing messages from NATS/JetStream to Mongodb so that the connector becomes bi-directional similar to what ClickHouse have: https://clickhouse.com/docs/en/engines/table-engines/integrations/nats#table_engine-redisstreams-creating-a-table?

If contribution is open, i can start and submit a PR..

damianiandrea commented 1 year ago

Hi @ahmed-adly-khalil,

Thanks a lot for your feedback, I'm glad you like the connector!

Regarding your question: I built the connector with the intent of simply sinking the data from MongoDB to NATS, but I think it would be quite cool to make it bidirectional (sourcing data from NATS to MongoDB). In fact, another user already proposed something similar.

Contributions are welcome! But since supporting bidirectional communication between NATS and MongoDB is not trivial, I invite you to share how you would implement that, and maybe discuss the approach together, before submitting a PR.

ahmed-adly-khalil commented 1 year ago

@damianiandrea Sounds good, what i had in mind is to use JetStream durable consumers https://docs.nats.io/nats-concepts/jetstream#consumers that can be created via yaml config to watch specific streams and sync data to mongodb, maybe that can happen in the client package of the connector

damianiandrea commented 1 year ago

@ahmed-adly-khalil, thanks for the feedback. I like your idea.

I was thinking of a configuration of this kind:

connector:
  # collections: collections to be watched, change events are published to a NATS stream (MongoDB -> NATS)
  # - ...
  streams: # streams to be consumed, messages are inserted in a MongoDB collection (NATS -> MongoDB)
    - streamName: "STREAM_NAME"
      subjects: 
      - "subject1"
      # - add more subjects here...
      dbName: "db_name"
      collName: "coll_name"

This would offer the most basic functionality: stream message -> collection.

A few points to consider (more will probably come up as we brainstorm):

Need to look out for cycles: imagine a config where we source messages from a NATS stream "COLL1" into a MongoDB collection "coll1". If we also have a config where we sink change events from the MongoDB collection "coll1" into a NATS stream "COLL1", we have a cycle.
In what format should the message payload from a NATS stream be persisted in a MongoDB collection?
Do we plan to just support insertions? What about updates and deletions? Maybe the stream subject can be used to determine if the message should be inserted/updated/deleted.

Just thinking out loud. Let me know what you think!

ahmed-adly-khalil commented 1 year ago

@damianiandrea these are great points, i love the config structure, let me draft a small pr and let's see how it goes, no worries if you tear it down,

for the cycles thinking we add a validation layer, maybe https://cuelang.org/docs/usecases/validation/ ?
format maybe can be part of the config, an enum with json only at start
rest of CRUD ops can follow, subject sounds good as operation, we can also use NATS message headers which can also help with more logic like idempotency?

damianiandrea commented 1 year ago

Hi, @ahmed-adly-khalil!

Forgive me for the very late response, life got in the way! :)

I think for now I'll close this issue, I'd like to focus on the current functionality of the connector, add more tests, measure its performance, and see what could be improved. After the current implementation is solid and works well, I might come back to this and pick up where we left off!

Thanks a lot for the feedback you gave so far!

damianiandrea / mongodb-nats-connector

NATS to Mongodb (vise versa) #3