search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 599 forks source link

Add support for Azure EventHubs #3255

Open elsesiy opened 5 years ago

elsesiy commented 5 years ago

Hi maintainers,

migrating an existing Storm project that relies on storm-eventhubs to use the Streamlet API fails as the interfaces (i.e. IRichBolt, IRichSpout, etc.) are not compatible.

It would be great to have native support for Azure EventHubs in Heron.

joshfischer1108 commented 5 years ago

Hi.

It looks like the Storm Eventhubs would be compatible with the Heron (Storm) Topology API or the ECO API. Would this work for you?

elsesiy commented 5 years ago

Hi,

I halted the effort to migrate our existing Storm topologies for now since all the available heron components such as container images, helm charts, documentation, etc. are outdated (still referring to 0.17.8). Happy to re-evaluate once 0.20.x release is final and the basic components work without much effort. The documentation in its current state lacks too much info on how to actually migrate an existing topology. In our case the Streamlet API seemed to be a good choice and seemed to be the only working solution to restrict the maximum amount of containers. A simple topology would constantly use 50 pods.

Happy to share the troubles if that helps.

joshfischer1108 commented 5 years ago

Interesting.. I'd like to follow up on the 50 pods topic soon.. I'm working to build us a new site right now that will be easier to maintain hopefully. In the mean time you can always build the docs locally by following the README in the website folder.

nwangtw commented 4 years ago

"seemed to be the only working solution to restrict the maximum amount of containers" I am not sure what it means. The low level API (which is pretty similar to Storm API) has a function to set number of stream manager which is the same as set the number of containers.

nwangtw commented 4 years ago

@elsesiy Also, I would really like to know your troubles and what are the main pain points. I guess one possible issue is that Heron supports a very old Storm API? Thanks in advance.