Redis sink: support for streams

[NicolasFloquet]

Hello!

Since redis is already supported as a sink, I feel that supporting streams could be a nice low hanging fruit feature to add.

Proposal

Suggested configuration:

sinks:
  my_sink_id:
    type: redis
    inputs:
      - my-source-or-transform-id
    url: redis://127.0.0.1:6379/0
    data_type: stream
    stream:
       name: 'stream-name'
       key: 'my-key'
       maxlen: 10

With such configuration, each log would be added to the stream as following:

XADD stream-name MAXLEN 10 * my-key <logdata>

[NicolasFloquet]

I am, by the way, eager to contribute for such feature. Please advise if you want me to start working on it :)

[spencergilbert]

It looks like the original issues around the redis sink included stream support - but I don't see any comments about why it was dropped, perhaps just scope reduction. We'd be happy to take contributions, I suspect it's small enough that it doesn't need an RFC but I'll defer to @jszwedko

[jszwedko]

Indeed, I'm not sure why it was dropped from the original PR, but agreed, it'd be great to have. I also agree the scope is small enough that I think just an implementation PR would be sufficient. Thanks for suggesting this @NicolasFloquet !

[jmagnuson]

Distilling the takeaways from #10172, the ask by @jszwedko is to directly map event fields to redis stream fields. E.g.:

{ "message": "hello world", "level": "DEBUG"}

becomes

xadd <key> * message "hello world" level DEBUG

This makes sense to me. One question I have would be how to handle fields with non-primitive types, like objects or arrays. E.g.:

{ "message": "hello world", "level": "DEBUG", "metadata": {"foo": "bar", "baz": 2} }

As far as I know, redis streams only support one layer of structure, while vector events can be arbitrarily structured. How should this be reconciled? Would metadata in this case just be encoded as a JSON string?

xadd <key> * message "hello world" level DEBUG metadata '{"foo": "bar", "baz": 2}'