search

frappe / event_streaming

Event Streaming for frappe
Other
28 stars 31 forks source link

Event stream timeout from function pull_from_node #9

Open koplingganda opened 8 months ago

koplingganda commented 8 months ago

Hi,

I just wanted to share with you an issue regarding the event stream. If we have a large amount of data from the Event Update Log it tends to hit an API in one go and can cause timeout. The most common timeout is from RQJob timeout and when I check the pull_from_node function, it uses the default queue which takes 300 seconds.

Is it possible if we can use custom worker for event stream, or maybe we can extend the timeout and batched the process of notify the consumer site?

Thank you in advance

koplingganda commented 8 months ago

Right now I override one function which is frappe.event_streaming.doctype.event_producer.event_producer.new_event_notification to be able to use long or custom worker queue