Add support for NATS Jetstream

[mfreeman451]

Feature Request: Add Support for NATS Jetstream in Proton/Timeplus

Use Case

Proton/Timeplus is recognized for its high-performance streaming SQL capabilities, largely powered by ClickHouse. To enhance its integration capabilities and messaging support, it would be beneficial to add support for NATS Jetstream. NATS Jetstream offers advanced features like persistence, at-least-once delivery, consumer offsets, and replayability, which are essential for reliable and scalable message streaming in complex, distributed systems. By integrating NATS Jetstream, Proton/Timeplus can be leveraged in more diverse environments, particularly where NATS is already an integral part of the messaging infrastructure.

Desired Solution

The ideal solution would be the implementation of a NATS Jetstream connector within Proton/Timeplus. This connector should be capable of both publishing to and subscribing from NATS Jetstream topics. Key features would include:

• Stream Ingestion: Ability to ingest data streams from NATS Jetstream, enabling real-time analytics on the streaming data.
• Fault Tolerance: Ensuring at-least-once delivery semantics to guarantee message processing in the face of network or system failures.
• Scalability: Support for NATS Jetstream scalability features, including partitioning and load balancing across multiple consumers.
• Ease of Configuration: Simple configuration setup for connecting Proton/Timeplus with a NATS Jetstream server, including stream and consumer configuration.
• Data Transformation and Enrichment: Capability to perform SQL-based transformations and enrichments on the ingested data from NATS Jetstream.

Alternatives Considered

Currently, integrating NATS Jetstream with Proton/Timeplus requires custom solutions or middleware, which can be complex and less efficient. Other messaging systems like Apache Kafka are supported, but they do not align with the infrastructure where NATS Jetstream is already in use.

Additional Context

The addition of NATS Jetstream support would significantly enhance Proton/Timeplus's appeal in environments where lightweight and high-performance messaging is crucial. It would also align Proton/Timeplus with modern cloud-native architectures that are increasingly adopting NATS Jetstream for messaging.

[jovezhong]

Thank you @mfreeman451 to add this feature request. It's certainly something we badly want to add to Proton. Just checking whether you may want to contribute code on this.

[mfreeman451]

would love to @jovezhong but C++ is outside of my wheel house.

[jovezhong]

No problem at all. Maybe we can invite external contributors to work on this and you can help us to review/test the integration. Let me setup a bounty on this. Our team right now are adding native ClickHouse support and will add more enhancements to the Kafka External Stream. If any one who are familiar with C++ and NATS to pick up this task, that'll be great

[jovezhong]

/bounty $300

[algora-pbc[bot]]

💎 $300 bounty • Timeplus

💎 $700 bounty • derekcollison

Steps to solve:

1. Start working: Comment /attempt #535 with your implementation plan
2. Submit work: Create a pull request including /claim #535 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Additional opportunities:

• 🔴 Livestream on Algora TV while solving this bounty & earn $200 upon merge! Make sure to have your camera and microphone on. Comment /livestream once live

Thank you for contributing to timeplus-io/proton!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @oliverqx	Jan 31, 2024, 2:46:10 AM	WIP
🟢 @Sambit003	Apr 2, 2024, 2:02:39 PM	WIP
🟢 @ayewo	May 20, 2024, 2:49:41 PM	WIP

[oliverqx]

/attempt #535

Algora profile	Completed bounties	Tech	Active attempts	Options
@oliverqx		TypeScript, Python		Cancel attempt

[hwinkel]

Has not clickhouse already a NATS integration ?

[yokofly]

Has not clickhouse already a NATS integration ?

no, clickhouse still does not fully support NATS Jetstream I guess? I find this PR still working in progress: https://github.com/ClickHouse/ClickHouse/pull/51281 full discussion here: https://github.com/ClickHouse/ClickHouse/issues/39459

[jovezhong]

We certainly can leverage some code of NATS integration in ClickHouse. Most likely it'll be a special Table Engine in ClickHouse. However in Proton, we prefer reading/writing NATS with External Stream, just like how it reads/writes Kafka, so that we can process latest events in NATS continuously, in the streaming fashion, instead of running a bounded query for the Table Engine multiple times to get the updates.

[hwinkel]

Has not clickhouse already a NATS integration ?

no, clickhouse still does not fully support NATS Jetstream I guess? I find this PR still working in progress: ClickHouse/ClickHouse#51281 full discussion here: ClickHouse/ClickHouse#39459

Hi,

OK; I see, Clickhouse has Core NATS but not acting as a Jetstream (durable) consumer.

[Sambit003]

/attempt #535

Options

• Cancel my attempt

[jovezhong]

Hi @Sambit003 , glad to see you pick up this task. Feel free to ask any questions here or in https://timeplus.com/slack

[Sambit003]

@jovezhong why this is?? No support for GNU Compiler Collections??

[yokofly]

@jovezhong why this is?? No support for GNU Compiler Collections??

yes . we do not support gcc, plz use clang16. https://github.com/timeplus-io/proton/blob/develop/BUILD.md

historical reason, and we use forked llvm repo. personally, llvm is more friendly for learning.

[ayewo]

I’d like to attempt this feature but probably wont be able to do anything substantive until next week or so.

/attempt #535

Algora profile	Completed bounties	Tech	Active attempts	Options
@ayewo	3 timeplus bounties + 17 bounties from 4 projects	TypeScript, Rust, Java & more		Cancel attempt

[Sambit003]

@ayewo hey, I'm working on this. So if I'm dropping it, I'll let you know. You can continue with my PR then.