redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.15k stars 841 forks source link

Jsonnet support in Benthos #707

Open MrSplinterRat opened 3 years ago

MrSplinterRat commented 3 years ago

Hello!

Could you consider adding Jsonnet (https://jsonnet.org/) in 'Processors' ?

  1. This is a very common Json-processing language, there are many production-ready libraries for it, it is often used in configuration and CI-CD pipelines
  2. There is a mature implementation on Golang from Google, https://github.com/google/go-jsonnet - it only remains to implement it in Benthos

Thanks)

Jeffail commented 3 years ago

Hey @MrSplinterRat, totally, that seems like a reasonable fit. We could even add this as a Bloblang function, maybe it would look like this:

root = jsonnet("""{
  person1: {
    name: "Alice",
    welcome: "Hello " + self.name + "!",
  },
  person2: self.person1 { name: "Bob" },
}""")

Which would allow you to do stuff like:

root = this.merge(jsonnet(file("./stuff/doc.libsonnet"))
MrSplinterRat commented 3 years ago

Using jsonnet from within Bloblang seems like a very nice and sane idea to me, and it will be useful in many cases. But along with this, I think it would be nice if Jsonnet also appeared in the list of 'Processors' as it is - similar to how it is now done for Awk, jq and Sql. Thus, it will be possible to use both from Bloblang-scripts and as an independent Processor. I see this as the best solution. Do you see it the same way?

illotum commented 3 years ago

I was under the impression the Processors API is subject to change in V4. Adding it to processors means more code to port.

Jeffail commented 3 years ago

That should be fine, it'll be easy enough to migrate the current style over for internal code so I wouldn't say this is blocked on v4.