matang28
commented
4 years ago Hi @weldpua2008, First of all thanks for your interest, this library is currently under development and it meant to replace our old logstash pipelines. We are currently using it in our staging environment and I believe that we will push it to production load this week.
About the stability, currently we don’t see any major issues regarding the steams themselves but as I said its not tested on production yet. When we finally push it to production and making sure it works properly I’ll add documentation (currently not priority because I might make breaking changes).
If you want the gist, My motivation was to create a simple yet powerful, highly-customizable, non-opinionated streams library for Go.
The library has the following concept: 1) Source, well data has to come from somewhere.... 2) Stream, created from sources allows you to map, filter and sink the source data. 3) Sinks, well data has to go somewhere eventually.... 4) Processors, this is a bit tricky to get at first, but processors responsible for taking elements from the source channel and move its through the stream. The built in implementations are:
- DirectProcessor, that simply run the element from source through the stream. One after one.
- BufferedProcessor, that uses micro-batches based on batch size and max time to wait.
If you want to see an example of all the concepts combined checkout this: https://github.com/matang28/go-streams/blob/master/example_test.go
And make sure to take a look on our connectors repo which currently has integrations to Kafka, couchbase, JSON, protobuf, clickhouse and redis is my next goal: https://github.com/matang28/go-streams-connectors
Dear @matang28,
I am searching for my jobs executor some library to deliver commands output (
stdout&stderr) to remote API or Redis.I found your repo and reugn/go-streams.
I wonder if your code is maintained and "stable" for use? And can you please add
README.md?