Synchronous Sending

[andrewgy8]

Hey all! Thank you so much for maintaining this fork!

The project looks super promising, and I think there is huge room for Faust in the ecosystem. I look forward to helping our where possible in the future.

One question I had: is there a simple way to publishing/sending synchronously?

We have a Django app with all sync code and we are not looking to migrate anytime soon. I have looked at the Django helpers async_to_sync, and thought about using them, but Im not sure of the downsides to using them are in the medium term. Perhaps it would skip batching messages to send to the broker?

On the consumer side, Im not so worried as using something like sync_to_async wont incur too much overhead.

Any thoughts on this? Thanks!

[wbarnha]

Regarding producing synchronously, there is the send_soon method defined in https://faust-streaming.github.io/faust/reference/faust.html#faust.Topic.send_soon. However, I do not recommend using it because I've observed my Faust applications become very unstable when too many synchronous blocking calls are made, including calls to send_soon. I believe this is a consequence of everything because abstracted as a service in mode where they must be periodically polled as a livecheck.

I do not have any experience using async_to_sync unfortunately. My honest opinion is that this project should be used with as few synchronous calls possible. There are countless instances of reports where their Faust consumers stop working for unknown reasons. I have had similar issues in the past- but as I reduced the number of synchronous blocking calls, I've seen less instability in general. I wish I had more time to examine this phenomenon further.

[andrewgy8]

Thanks for the quick response @wbarnha ! I appreciate the advice on the sync calls.

Yeah, the lost messages do sound concerning, especially for mission critical ones. Your instability of consuming messages, did that come from using the consumer as a synchronous process (not sure this is even possible with Faust)? Or using sync code in an async consumer process?

And what kind of load did you see consumers acting up?

[wbarnha]

Thanks for the quick response @wbarnha ! I appreciate the advice on the sync calls.

Yeah, the lost messages do sound concerning, especially for mission critical ones. Your instability of consuming messages, did that come from using the consumer as a synchronous process (not sure this is even possible with Faust)? Or using sync code in an async consumer process?

I would not advise using synchronous consumers with Faust. What library are you trying to use?

And what kind of load did you see consumers acting up?

Hard to say, many users, including myself, have seen instability at a variable number. I apologize for the lack of clarity as I too am trying to understand the root cause.

[andrewgy8]

Hard to say, many users, including myself, have seen instability at a variable number. I apologize for the lack of clarity as I too am trying to understand the root cause.

Do you still consider Faust to be production ready? I really like the concept and ergonomics around it, but it is a bit worrying. I suppose there are different use cases depending on criticality.

[wbarnha]

Hard to say, many users, including myself, have seen instability at a variable number. I apologize for the lack of clarity as I too am trying to understand the root cause.

Do you still consider Faust to be production ready? I really like the concept and ergonomics around it, but it is a bit worrying. I suppose there are different use cases depending on criticality.

Yes,. it's definitely production-ready. Many performance issues stem from people inserting too many synchronous blocking functions into their applications.

[andrewgy8]

Thanks for the guidance! I look forward to giving it a go!