RFC-1: Do we need a Message Queue

[WilfredAlmeida]

The execution order when the system starts will be:

1. Fetch the latest processed transaction signature on a tree from the database, let's call it lastProcessedTxSignature
2. Pull transaction signatures from the latest transaction to lastProcessedTxSignature from the chain
3. Pull and process each transaction data
4. Initiate the WebSocket listeners for the trees

Suppose:

• The program begins at the time T1
• Steps 1-3 take 10 seconds and complete at the time T10
• Step 4 begins at T11 and listens for transactions on the tree as they happen

The transactions between time T1-T10 will not be processed and result in an incorrect state in the database

Solutions:

1. Starting both the synchronous processing of transactions and Websocket listeners on trees at the same time

   • This won't work as the synchronous processing, and WebSocket listeners will upsert the database together, and the state will be incorrect

2. Setup a message queue and worker threads/processes to process transactions

With this approach, the flow will be:

1. The main thread starts with both synchronous transaction processing and WebSocket listeners
2. Both push their respective transaction signatures into their respective queues, let's call them synchronousTxsQueue and websocketTxsQueue
3. A worker thread picks transaction signatures from these queues and processes them
4. The worker thread will have the logic

   if synchronousTxsQueue.hasNext()
   processTransaction(synchronousTxsQueue.deQueue)
   else
   processTransaction(websocketTxsQueue.dqQueue)

Suggested Message Queue:

1. Kafka: Supports consumer notification, replication, and retries, and can handle scale. Will be future-proof
2. Redis: Metaplex DAS already has Redis which is used for data streaming. A similar stream can be added to it

Questions:

1. Is approach 2 good enough?
2. If yes, which queue is good to use?

[simplysabir]

The execution order when the system starts will be:

1. Fetch the latest processed transaction signature on a tree from the database, let's call it lastProcessedTxSignature
2. Pull transaction signatures from the latest transaction to lastProcessedTxSignature from the chain
3. Pull and process each transaction data
4. Initiate the WebSocket listeners for the trees

Suppose:

• The program begins at the time T1
• Steps 1-3 take 10 seconds and complete at the time T10
• Step 4 begins at T11 and listens for transactions on the tree as they happen

The transactions between time T1-T10 will not be processed and result in an incorrect state in the database

Solutions:

1. Starting both the synchronous processing of transactions and Websocket listeners on trees at the same time

• This won't work as the synchronous processing, and WebSocket listeners will upsert the database together, and the state will be incorrect

2. Setup a message queue and worker threads/processes to process transactions

With this approach, the flow will be:

1. The main thread starts with both synchronous transaction processing and WebSocket listeners
2. Both push their respective transaction signatures into their respective queues, let's call them synchronousTxsQueue and websocketTxsQueue
3. A worker thread picks transaction signatures from these queues and processes them
4. The worker thread will have the logic

if synchronousTxsQueue.hasNext()
    processTransaction(synchronousTxsQueue.deQueue)
else
    processTransaction(websocketTxsQueue.dqQueue)

Suggested Message Queue:

1. Kafka: Supports consumer notification, replication, and retries, and can handle scale. Will be future-proof
2. Redis: Metaplex DAS already has Redis which is used for data streaming. A similar stream can be added to it

Questions:

1. Is approach 2 good enough?
2. If yes, which queue is good to use?

1. Is approach 2 good enough?

Yes, it addresses the main issue of ensuring all transactions are processed in the correct sequence without loss or duplication. It provides a scalable and reliable architecture that can adapt to varying loads and network conditions.

2. If yes, which queue is good to use?

Ideally Kafka is used but if metaplex is already using Redis and it aligns with us, should go with Redis. as Kafka can be complicated sometimes

[WilfredAlmeida]

We've decided to move ahead with an in-memory queue. Please refer to the documentation for more details