Users can be notified when the indexer ingests documents/transactions

[spacegangster]

(original message below)

We'd like users to be able to receive notifications from the indexer (and other areas, in the future) so that modules can react to transactions flowing through the Crux node. Potential use cases to consider:

• Collecting metrics (#384)
• Subscribing to indexed transactions to update downstream consumers
• Re-publish the events to Kafka topics.

Potential solution - event bus:

• must be very light weight. it shouldn't be slow, but also not complex
• it should publish the events async, potentially in a thread per subscriber module (not subscription).
• events may be split by topics, or the filtering could simply be done by the listener.
• the events need to adhere to specs - a few keys might be mandatory, like crux.event-bus/topic or similar.

---

A “hook” for processed txes would ease both user subscriptions to db updates and our work with #161 . It can be a hook, an event or a signal. In its most minimal form it's just a callback accepting tx meta #:crux.tx{:tx-id 1, :tx-time 1}, so users could leverage that when working with tx-log.

Perhaps, an internal hook for accepted non-processed transactions could also ease development of new tx-log consumers, maybe even streamline the existing ones; but here I'm not so sure.

This issue is here to accumulate more thought and feedback from the core devs and the community.

[fjolne]

+1. That would greatly aid our transition from Firebase RealtimeDB to Crux for our custom chat service. What we basically need is a pub/sub interface to notify users of new messages, and as far as I understand currently the only way to do it with Crux is to manually subscribe to Kafka commit log.

[mhuebert]

I am interested in similar use-cases - ability to monitor transaction logs, and send 'invalidation' messages to subscribed clients when particular entities or entity-attribute patterns change.

One could also build something like firebase database-triggered cloud functions on top of this.

[jarohen]

(note to self, and @t-taylor - this might well be another application of the topology middleware we're currently considering)