search

mage-ai / mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.
https://www.mage.ai/
Apache License 2.0
7.74k stars 732 forks source link

feat: Add support for NATS #3971

Closed mfreeman451 closed 10 months ago

mfreeman451 commented 10 months ago

Is your feature request related to a problem? Please describe.

I have been using Mage-AI for data processing and automation tasks and have found it extremely efficient and user-friendly. However, in my current workflow, I heavily rely on NATS as a messaging system for real-time, distributed communication. The lack of integration between Mage-AI and NATS limits my ability to seamlessly manage data pipelines and real-time data streams. Currently, to work around this, I have to employ additional intermediate services or custom scripts, which complicates the architecture and introduces potential points of failure.

Describe the solution you'd like

I would like Mage-AI to add native support for NATS. This integration would ideally allow Mage-AI to subscribe to NATS topics, process messages as they arrive, and possibly publish results to other NATS topics. This feature would streamline real-time data processing workflows, enabling Mage-AI to interact directly with NATS for efficient data ingestion and event-driven processing.

Describe alternatives you've considered

As an alternative, I've considered using other messaging systems that are currently supported by Mage-AI. However, NATS is particularly suited for my use case due to its high performance, scalability, and ease of use. Another alternative would be to develop a custom service to bridge Mage-AI and NATS, but this adds complexity and maintenance overhead.

Additional context

NATS is widely used in cloud-native environments for its simplicity and performance in handling message streams. Integrating NATS with Mage-AI would not only benefit my use case but could also enhance Mage-AI's appeal to users in similar cloud-native and real-time processing environments. This feature could open up new possibilities for Mage-AI in event-driven architectures and improve its utility in modern data processing pipelines.

https://nats.io/

wangxiaoyou1993 commented 10 months ago

Thank you for the feature request @mfreeman451 ! When do you need this feature by?

mfreeman451 commented 10 months ago

We could use it as soon as possible. I took a look at the kafka integration, it doesn't look that complicated and I may try and submit a PR myself.

wangxiaoyou1993 commented 10 months ago

We could use it as soon as possible. I took a look at the kafka integration, it doesn't look that complicated and I may try and submit a PR myself.

Oh, that'll be great. Here is the doc for adding a streaming source: https://docs.mage.ai/contributing/backend/streaming/sources-and-destinations#add-a-new-source

ddecaro11 commented 2 weeks ago

Hello, I would be interested in adding credentials file authentication to NATS jetstream streaming source, would that be possible though a PR?

mfreeman451 commented 2 weeks ago

Hello, I would be interested in adding credentials file authentication to NATS jetstream streaming source, would that be possible though a PR?

I think, especially if you are following the projects contribution guidelines, PR's are always welcome.