Support NATS Streaming Server (stan)

[digitalsanity]

I would like to use NATS Streaming Server (aka stan -- https://github.com/nats-io/nats-streaming-server ) as my logging/analytics back-end. It uses a similar protocol to the existing NATS plugin but has a few additions/changes to support additional desirable features.

Reference: A golang NATS Streaming Server client is available @ https://github.com/nats-io/stan.go

Steps: Clone the existing nats plugin, replace connector code, review methods, arguments, configuration options.

Purpose: NATS is lightweight (~30MB RAM) and ideal for streaming analytics data but the data is transient.

NATS Streaming Server is a lightweight addition (~10MB RAM) that allows for disk-backed, rewindable streams tracked per subscriber with at-least-once delivery. It runs on the same port/connection as NATS but uses a different client/protocol.

I am currently using a workaround of running a separate "replicator" service that copies NATS subjects to NATS Streams. I was able to use fluent-bit tail (throttled at ~1000 records/second) to import~5.6GB of nginx json access.log data (~5 million json records) with very little CPU/memory usage. The replicator: https://github.com/nats-io/nats-replicator

NATS Streaming Server is an ideal partner for event-sourcing Fluent Bit inputs.

[edsiper]

@wallyqs would you please point me out to the Protocol spec?

[digitalsanity]

Sorry - I was thinking fluent-bit used golang plugins .. the nats.c client has a stan (streaming) example:

https://github.com/nats-io/nats.c/tree/master/examples/stan

[edsiper]

Pure, fast and optimized C code :)

[wallyqs]

@edsiper the protocol spec is here internally it is using NATS Requests/Responses with the payloads encoded with protocol buffers, but the STAN library that @digitalsanity pointed out should do all of that for you: https://github.com/nats-io/nats.c/blob/master/examples/stan/pub.c#L79-L96

[edsiper]

thanks @wallyqs , I will take a look

[jhonatanmorais]

Hi guys, is there any working in progress about it?

[rsilva-rs]

Hi @edsiper ! Thanks for taking on that it surely also help others.

@digitalsanity pls, how does it work that workaround you mentioned using NATS Replicator.. Fluent-bit -> NATS -> NATS-Replicator - NATS-Streamming? Do you mind sharing your Replicator config?

[digitalsanity]

@rsilva-nk

I cloned this repo:

https://github.com/nats-io/nats-replicator

Using a config like this:

`reconnectinterval: 5000,

logging: { Time: true, Debug: true, Trace: false, Colors: true, PID: false, },

monitoring: { httpport: 9090, }

nats: [ { Name: "nats", Servers: ["172.17.0.1:4222"], ConnectTimeout: 50000, MaxReconnects: 50000, ReconnectWait: 15000, } ]

stan: [ { Name: "stan", NATSConnection: "nats", ClusterID: "test-cluster", ClientID: "replicator-one" } ]

connect: [ { type: NATSToStan, id: "alpha", IncomingSubject: "in", IncomingConnection: "nats", OutgoingChannel: "foo", OutgoingConnection: "stan", }, { type: NATSToStan, id: "alpha", IncomingSubject: "my_cpu", IncomingConnection: "nats", OutgoingChannel: "my_cpu.stream", OutgoingConnection: "stan", }, { type: NATSToStan, id: "alpha", IncomingSubject: "abc.abc1.access-log-test", IncomingConnection: "nats", OutgoingChannel: "stan.abc.abc1.access-log-test", OutgoingConnection: "stan", },

], ` The IncomingSubject stream gets copied to the OutgoingChannel stream. You can then use a "stan" client to subscribe to the outgoing channel and rewind/etc.

I am launching with this:

docker run -v /work/src/nats-replicator/conf/replicator.conf:/conf/replicator.conf -p 9090:9090 "nats-io/nats-replicator" -c /conf/replicator.conf

This expects your NATS to be running on 172.17.0.1:4222.

So now whenever fluentbit logs into abc.abc1.access-log-test NATS, it gets automatically forwarded to stan.abc.abc1.access-log-test STAN using the stan instance running on that same NATS. Replicator acts as a standard subscriber to NATS and publisher for STAN.

*STAN will be replaced by LiftBridge eventually .. but there were some issues last I tested.

[rsilva-rs]

@digitalsanity thank you so much for sharing.

I ended up using Fluentd instead with a NATS Stan community plugin. However, still considering to use Fluent-bit as collector and Fluentd as proxy based on Forward plugin for interoperability between Fluent-bit and Fluentd.

In any case, NATS Stan plugin for Fluentd: https://github.com/hc-chien/fluent-plugin-nats-streaming