search

maki-nage / makinage

Stream Processing Made Easy
https://www.makinage.org
MIT License
39 stars 1 forks source link

Good replacement for faust for non ML project? #8

Closed ericsouza closed 3 years ago

ericsouza commented 3 years ago

Hi, I'm looking for a python framework to deal with kafka events with async consumers. My needs is not for Machine Learning, but just receive kafka messages, call external APIs and save some data in redis/postgres. Do you think this project is a good choice for that? Faust-streaming have a lot of cool stuff (windowed, tasks, timers, monitoring) but the project seems kind dead

MainRo commented 3 years ago

The focus of maki-nage is a little bit different than Faust (BTW building maki-nage on top of Faust was one of my early options). As you said, maki-nage is more focused on data-science use-cases. This means that we try to provide higher-level APIs and less flexibility than Faust. Typically you will not be able to call async IOs in a maki-nage application for now. This may be possible in the future because we already have some similar needs, but this is not a priority for now.

However, if you need things like windowing operations, you can also go with your own AsyncIO application, and use rxsci for the data transforms. Bridging AsyncIO and rxsci is relatively easy both on the source and sink sides.

ericsouza commented 3 years ago

@MainRo Oh, I see. Thanks for the answer!!