search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
7.99k stars 789 forks source link

Support singer.io schema #716

Open gedw99 opened 3 years ago

gedw99 commented 3 years ago

Its python

Some has written a golang schema for it. https://github.com/pbegle/singer-go

Here is one implementation of it: https://github.com/transferwise/pipelinewise doc: https://transferwise.github.io/pipelinewise/ k8 operator: https://github.com/dirathea/pipelinewise-operator

there are so many useful taps, and it makes it easy to do those integrations that takes huge amounts of code checked for data changes not being ordered. It seems to use a log based approach.

Is can do CDC off databases also, but also it seems other APIS.

Jeffail commented 3 years ago

Hey @gedw99, it'd be interesting to see if we can integrate with this. From the looks of it we'd probably need to add this as some sort of processor.

gedw99 commented 3 years ago

sorry about not responding. I got heavily side tracked with other stuff.

to manage expectations, i realised that as cool as Singer is, its does not meet my needs.

Please close or leave open @Jeffail