search

cube2222 / octosql

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
Mozilla Public License 2.0
4.75k stars 201 forks source link

nats driver ? #264

Closed gedw99 closed 2 years ago

gedw99 commented 2 years ago

I just stumbled on this repo. It looks alot like Differential dataflow pattern ?

I was wondering if NATS is considered ? Kafka i saw is there.

Nats Jetstream can do deduplication i read btw now.

cube2222 commented 2 years ago

Hey!

Yeah, Differential Dataflow is quite similar if looked at from a distance. The technical details differ quite a bit though.

I'm currently working on a ground-up rewrite of OctoSQL on the redesign branch so I definitely won't add it in the close future. Long term NATS totally makes sense though and should be very easy to support, so I'll leave the issue open.

gedw99 commented 2 years ago

Thanks for update

i will check out the redesign .

if there is a design intent doc let me know

cube2222 commented 2 years ago

Hey, there's no doc as the project doesn't accept contributions for the timespan of that redesign (9 months from now)

However, there will be a plugin system for easy development of new datasources included which should be perfect for this.

gedw99 commented 2 years ago

ok thanks.

I am using materialize for now just to get going.

You probably know these links, but in case you dont.

https://github.com/MaterializeInc/materialize https://github.com/TimelyDataflow

gedw99 commented 2 years ago

Me again

looks like the resins is now established

i just read the docs in the readme and am still a little confused on the way it does watermarking as a way to feed data inputs into outputs .

You use input and output terminology as opposed to sources and sinks, so try to wrap my head around it still.