Registry WebHook processing Performance Issue

[t83714]

Registry WebHook processing Performance Issue

Registry is responsible for notify webhooks for any interested new events.

Currently, registry will generate complex event fetching SQL based on the webhook's interested, let DB to pick all relevant events and sent to the web hook.

Depends on the web hook's config, the generated SQL might be complicated enough (as it might involved sub queries & multiple table joins) to cause performance issue to the DB.

To avoid that, we probably should try the alternative solution:

• Let DB fetch a page of event (e.g. 1000 events) without any filters
• Run through all events locally based on webhook config one by one

This might look like inefficient. But once the web hook is up to date, there won't be too many events generated per seconds and max no. of events to be filtered is always up to 1000 events. Compared with the current solution, the DB might (depends on query plan & optimisation) always have to go through the whole event logs to pick up the events.

[t83714]

closed via https://github.com/magda-io/magda/pull/3409