search

castorm / kafka-connect-http

Kafka Connect connector that enables Change Data Capture from JSON/HTTP APIs into Kafka.
https://castorm.github.io/kafka-connect-http/
Apache License 2.0
124 stars 56 forks source link

HTTP Streams #126

Open novucs opened 3 years ago

novucs commented 3 years ago

Is your feature request related to a problem? Please describe. I haven't seen an option on this connector for infinitely streaming data. An example could be found here on the Twitter API, that sends a gzipped stream of tweets: https://developer.twitter.com/en/docs/twitter-api/enterprise/powertrack-api/api-reference/powertrack-stream#Stream

Describe the solution you'd like Perhaps a boolean option upon creation of the connector to say that were connecting to an infinitie stream of data. Additionally maybe another option to support converting data formats such as gzip.

Describe alternatives you've considered This appears to be an unmaintained version, only supporting an older Twitter streaming API: https://github.com/jcustenborder/kafka-connect-twitter

Additional context I can see a lot of people using this connector if it's able to support HTTP streaming, such as the Twitter APIs

devarismeroxa commented 3 years ago

This would be great. 👍🏾

castorm commented 2 years ago

Hi @novucs,

Thank you very much for filing this request.

I see the value of supporting such feature, although I'm not sure there is much overlap between the current implementation of this plugin and what you suggest. This plugin was originally intended for traversing a paginated HTTP source, so I wonder whether this project is a good fit for this or it'd be best a new plugin altogether.

Anyway, contributions are always welcome!

Best regards.