At this moment, the indexer expects the registry API to search for all relevant events through aspect reference links and send over a list of updated datasets rather than raw events.
This might lay too much pressure on the registry API & database.
Instead, we should:
Indexer listens to the raw events instead, and the registry will only send over raw events
The indexer should combine the raw events (e.g. events for the same record in the same page only trigger one index action) and re-construct the required dataset data from the raw events.
Allow webhook to specify desired event page size in config
Indexer Performance Improvement
At this moment, the indexer expects the registry API to search for all relevant events through aspect reference links and send over a list of updated datasets rather than raw events.
This might lay too much pressure on the registry API & database.
Instead, we should: