search

reugn / go-streams

A lightweight stream processing library for Go
https://pkg.go.dev/github.com/reugn/go-streams
MIT License
1.89k stars 158 forks source link

DSL / Config for describing the Flow #62

Open gedw99 opened 1 year ago

gedw99 commented 1 year ago

We could describe the flow using a file dsl / idl. A type of config fike.

Mythen the runner loads this config file.

It’s almost like how Benthos works though .

It would be useful to write a Comparison with Benthos in the readme too because there is Carey over in sone ways

rupurt commented 1 year ago

I feel like streaming SQL syntax like Flink, Materialize, RisingWave etc... would be a better fit as that is super common in the streaming space and already has pre-established syntax & constructs.

gedw99 commented 1 year ago

That would be fantastic. With CDC pattern for data source change events

gedw99 commented 1 year ago

Is there any start on this in the code ? Would be interested in helping.

checkout appcash pranadb on GitHub btw. Has sone similarities to your ideas .. gotta run

gedw99 commented 1 year ago

Also if you have links to explain sql like streaming it would be cool.

materialize is a rust project that sort of does this against postresql db

gedw99 commented 1 year ago

https://github.com/reugn/go-streams/blob/3771cd778875ede809125c8053be06406ad6fd95/examples/nats/main.go#L33

Would be good to see how that can be a SQL like thing .

the lever and parse could be like steam pipe project maybe that uses HCL