FEEDBACK WANTED: New `bloblang` processor

redpanda-data / connect

Fancy stream processing made operationally mundane

https://docs.redpanda.com/redpanda-connect/about/

8.14k stars 838 forks source link

FEEDBACK WANTED: New `bloblang` processor #439

Closed Jeffail closed 4 years ago

Jeffail commented 4 years ago

Hey everyone, there's now a new bloblang processor on offer in the latest Benthos release 3.13.0. Blog post here for more context: https://www.benthos.dev/blog/2020/05/10/bloblang-beta.

There's still room to tweak the language, and given the scope of Bloblang I'm looking for people to try it out and give me feedback. This is important as I ultimately want to deprecate a lot of processors in favour of just this one.

More information about the language can be found at: https://www.benthos.dev/docs/guides/bloblang/about.

Feel free to dump thoughts/questions here, on the gitter channel or even just @ me on twitter.

cludden commented 4 years ago

First off, nice work on this, I've been wanting something like bloblang for a while, really simplifies pipelines!

Question/Bug? when using the bloblang processor with an http_server input, it seems the result store is discarded or not copied to the new message, at least that's my interpretation of this error:

Failed to send message to sync_response: result store not found within batch context

Jeffail commented 4 years ago

Thanks @cludden, that's definitely a bug. Which version of Benthos did you see that with? And can you give me any details about the pipeline? I tried it out and it seems to work with master so hopefully this has already been fixed.

cludden commented 4 years ago

Ignore my question above, it was due to me not understanding how the sync_response processor works in conjunction with filter

Jeffail commented 4 years ago

Removing the BETA flags and releasing https://github.com/Jeffail/benthos/releases/tag/v3.15.0, this means no more breaking changes but I'm still interested in feedback.

cludden commented 4 years ago

some things I'd love to see in bloblang, especially now that some conditions/processors are on the chopping block:

a way to extract data from arbitrary text, like the old regexp_expand text operator, or idml's match method
if expressions (similar to idml or jsonnet)
a way to check the data type of an expression, similar to jsonnet's std.type
```
root = match {
this.type() == "array" => deleted()
}
```
a way to express multi-line strings, similar to yaml's | or jsonnet's |||

Jeffail commented 4 years ago

Thanks @cludden, good stuff. Some of those features are already on my mental roadmap, might get them in this weekend.

zlozano commented 4 years ago

Huge fan of bloblang. I have used it quite a bit recently. One thing that has come up is the desire for short-circuit/McCarthy evaluation for the lhs and rhs operands in a && and || operation. A concrete use-case where this is handy is when conditions involve the existence of metadata.

bloblang: meta().contains("foo") && meta("foo") == "bar"

Today, this results in an error metadata value not found since both operands appear to be eagerly evaluated.

Jeffail commented 4 years ago

Thanks @zlozano, that should be quick and easy to fix, I'll look at it this weekend: https://github.com/Jeffail/benthos/issues/464

Jeffail commented 4 years ago

Closing for now, if anyone has feedback go ahead and open a new issue or get in touch.