Websocket Streaming API in antelope_memento_api

[cc32d9]

The new API will allow the user to connect to the websocket endpoint and subscribe to events for multiple blockchain accounts.

The wsebsocket service needs to listen on a separate TCP port, so a new environment option is required. socket.io protocol is used for communication.

The connection serves only one request at a time. A subsequent call on startStreaming will cancel the previous streaming request in this socket.

startStreaming arguments:

• accounts: array of blockchain account names
• start_block: optional starting block number. If omitted, the streaming starts from the current head block.
• irreversible: optional boolean. If set to true, only updates from irreversible blocks are streamed.

The server sends events of the following structure:

• type: string event type, valid values: fork, trace
• block_num: block number
• data: transaction trace or null

If the starting block is lower than last irreversible block, the API needs to retrieve the historical data from RECEIPTS and TRANSACTIONS tables in chunks and feed them tot he client. The size of the chunk is determined as follows: we get the number of traces within the next 100 blocks (the number is configurable), and if it's more than 100 traces (configurable), the block range is reduced proportionally.

/* first, estimate the number of traces */
select count(distinct seq) from RECEIPTS where 
  receiver in ('atomicassets', 'eosio') and block_num >= 246026319 and block_num < 246026419;

/* if the resulting number is more than 100, take a lower end block. In this example, if the returned number is 300, 
the end block would be 246026319+33 = 246026352 */

select block_num, trace from TRANSACTIONS 
join (select distinct seq from RECEIPTS where 
  receiver in ('atomicassets', 'eosio') and block_num >= 246026319 and block_num < 246026352) R 
on TRANSACTIONS.seq=R.seq order by TRANSACTIONS.seq;

Once it reaches the last irreversible block, it should switch to querying EVENT_LOG. First it determines the starting ID, using the last processed block number:

select max(id) from EVENT_LOG where block_num=246373737;

Then it starts retrieving the records from EVENT_LOG in chunks of 100 (configurable) traces:

select id, block_num, data from EVENT_LOG where id>37130 and id <= 37230;

Once it reaches the position where common dispatcher is retrieving the data, the stream switches to the common dispatcher.

The common dispatcher remembers the last processed ID and retrieves a new chunk of data every 500ms in chunks of 100 (configurable) events with the same query as above. It then analyzes every trace and determins which clients should receive it.

While walking the EVENT_LOG, the server should look at the fields and determine if the trace needs to the sent to the client, as follows

1. in trace object, loop through action_traces. Within each action trace:
2. match receipt.receiver against the list of target accounts;

As soon as one of the matching criteria is true, abort looping through the action traces and push the trace to the client.

If there are no connected clients, the API does not need to scan the EVENT_LOG.

If a client is too slow in consuming the stream, the server should switch from head block back to scanning EVENT_LOG specifically for this client. If the scanning delays behind the last irreversible block, the server should switch to scanning RECEIPTS and TRANSACTIONS.

[cc32d9]

If the stream was requested with irreversible=true, the sever walks along RECEIPTS and TRANSACTIONS only. Before querying the traces, it needs to check if the last irreversible block has advanced:

select MAX(irreversible) from SYNC

If the value is higher than the one previously known, it fetches the traces as described above.

[cc32d9]

While walking through the irreversible history, the socket produces only trace events.

Once it crosses the Last Irreversible Block (LIR, the irreversible field in SYNC table), it starts reading events from EVENT_LOG table, and keeps sending trace events to the websocket client, checking the transaction contents and deciding if the transaction is related to the subscriber or not.

If EVENT_LOG contains a fork event, the websocket client should receive the event of type fork with the forked block number.

Once the reader reaches the head block, it should read new transactions from EVENT_LOG only once, and see which websocket clients need to receive the message: the trace events are filtered by the accounts subscription, and fork events need to be sent to all clients.