Cleanup and visibility enhancement

[sbrossie]

This task is to track some enhancements we could do on 0.24.x release

[sbrossie]

Can we simplify the reaper logic to only look at one date

NotificationQ

The current logic looks at both processing_available_date and effective_date. We expect the clause processing_available_date <= now to come prior effective_date <= reapingDate, therefore making the later condition the one that drives the reaper.

PersistentBus

The current logic is similar except the effective_date for the bus is the created_date.

Note that for inflight queue, processing_owner is null so processing_available_date <= now is never even considered.

=> Based on this, it seems we could remove processing_available_date altogether along with the claim time as this is only used for the reaper and we propose to simplify it.

[sbrossie]

Can we add some information about processing time?

One challenge here is to define what we want to measure:

• Time from when entry was fetched as ready to when it moved to history table - this will not represent the true processing time, the larger the batch we fetch the larger the time we would get
• Real time it took to go through all handlers - harder to measure

Implementing this means we would need a new field in the history table.

Additionally it would be great to add a created_date for the history table to check when this entry was processed (and for instance track whether this was processed late - we would keep the created_date from when entry was inserted.

[sbrossie]

Can we link REAPED entries - right now, a new entry is inserted with the same event_json, so unless we look for entries with same event_json, we can't really figure out what happened?

[sbrossie]

Can we provide a mechanism to pause the reaper - e.g as we insert a lot of entries at once such as during a migration, and where default settings don't make sense?

[sbrossie]

Can we easily export the runtime config for the queue - bus + notification ?