search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.67k stars 1.39k forks source link

S3 Compatibility for NATS Object Store #4871

Open wpnpeiris opened 9 months ago

wpnpeiris commented 9 months ago

Proposed change

Introduce a new module to NATS platform, which acts as a gateway for NATS object store. The new module, (let’s call it nats-s3-gateway, or nats-s3) will be a NATS client (JetStream client) but expose a REST API that adheres to the AWS S3 protocol. nats-s3 will be its own service, which abstracts away integration with NATS object store and provides S3 protocol implementation. See the PoC implementation here.

Use case

The S3 API is kind of a de facto standard for accessing Object Storage systems. Many object storage systems such as MinIO, SeaweedFS, JuiceFS, etc support S3 protocol. Having S3 protocol support NATS object store enables integration over HTTP.

Contribution

Yes, we are intending and interested in contributing to NATS. See the PoC implementation here.

ripienaar commented 9 months ago

We are currently working on a HTTP gateway for various aspects of NATS and for sure S3 compatible API would be a great addition.

Really great that you did a POC of this, we'll take a look when we get to thinking about the object store aspect of a above

kitobelix commented 9 months ago

Hey wpnpeiris! Why not also using nats (jetstream) as metadata store?

rducom commented 7 months ago

Considering the scalability and bandwidth capabilities of S3, would it be possible to leverage S3 not only as an API but as a backend storage solution for NATS ?

This approach could offer two operational modes: 1) synchronously storing topics entirely on S3, and 2) using S3 as asynchronous cold storage for long-term or infinite topic retention, maintaining a local hot cache.

Adopting S3 as a primary backend could significantly enhance scalability compared to traditional storage and potentially render NATS nearly stateless (without local disk persistence), as seen with similar projects like Warpstream's adaptation of Kafka.

derekcollison commented 7 months ago

We are looking into many options, including that one. Most folks struggle with the latency, at least in the NATS.io ecosystem. They expect very fast response times both for ingress and egress from JetStream.