Implement redundant listener.

[debendraoli]

This will help to counter missed events.

• Subscribe as many rpcs that are provided.
• Track event from each clients, detect and reconnect unhealthy client.
• Fallback to polling when there are 0 subscription and try to recover.

---
title: Relayer Redundancy
---
flowchart TD
    A[Start] --> B(Starup tasks)
    B --> C(Poller) --> H
    B --> D{Listener}
    D --> | Subscription | E[RPC-1]
    D --> | Subscription | F[RPC-2]
    D --> | Subscription | G[RPC-3]
    E & F --> H[Events]
    G --x H
    G --> | Reconnect | D
    H --> | Process | I(RPCs) <--> | Healthy? | J(Execute Queue)
    D --> | No subs? | C

[bcsainju]

Different Fetching Strategies

Provides multiple strategies to chhose from for any chains (Currently only implemented for evm chains)

Framework Changes

Height Sync

• For maintaining height sync for multiple strategies(especially redundant ws), the height sync functionality has been changed.
• The height save functionality is made available to chains as well. The framework height sync has extra checks added which will chek if the backlogProcessing flag is set true or not in the chains.
• Chains which want to control the height sync on its own can set the flag as True and then sync height on its own or disable temporarily during backlog processing of missed blocks.

Strategies Implemented:

Fetching Strategies

1. Websocket

• This is the current preferred implementation to get events.
• During chain startup, ws clients are started listening from latest height and another process starts to sync the missing block heights.

flowchart TD
    A[Relayer] -->|Load Configs| C(Start)    
    C -->|Ws| D[Listen to events via websockets]
    C -->|poll| E[sync missing blocks]

2. RPC poll

• For chains not supporting ws or reliable ones, we can use this strategy to fetch events.
• It sill simply query the RPC every x seconds to get events and update the latest height from the chain itself.

flowchart TD
    A[Relayer] -->|Load Configs| C(Start)    
    C -->|poll| E[sync blocks every x seconds]

Redundancy Strategies

Redundancy strategies are only applicable for websocket fetching strategy and not for the RPC poll as the poll is costly,slower but deterministic.

1. rpc-verify

• This strategy uses the RPC request alongside the websocket itself.
• After every x minutes, if the websocket is not producing any events, it will fire a RPC call to ensure events were empty during the lastSavedHeight and the latestHeight,
• If no events are found from the RPC request as well, then the height is saved and nothing more is done.
• If the RPC finds event during the verification phase, the events are queued and the websocket connection is refreshed assuming some error has occurred.The height is updated to the last block fetched during the verification.

flowchart TD
    A[Relayer] -->|Load Configs| C(Start)    
    C -->|Ws| D[Listen to events via websockets]
    C -->|poll| E[sync missing blocks]
    D -->|rpc-verify| X{ws events?}
    X --> |Yes| Y[do nothing]
    X --> |No| Z{verify with rpc if \n events are empty}
    Z --> |yes| M[return]
    Z --> |no| N[process events\nreset ws]
    N --> D

2. WsRedundancy

• This strategy changes the fetching strategy as well since it needs to listen to multiple websockets.
• Multiple clients are created during initialization. The value in the existing config for rpcurl and wsurl will be the main client used for all other purposes. The new endpoinst which will act as redundancy will be passed in the config under new list parameters and initialized upon provider creation.
• An internal LRU cache is created which will be used to hold packets which haven't reached quorum before forwarding to the main relay processing channel.
• Also, internal channel will be used to transfer packets from multiple websockets and not the main blockInfo channel itself.
• Each listener will send height update periodically and the chain will update the last processed height with the minimum height reported by the listeners.
• Whenever a packet arrives from the chain, it will be checked if it has reached quorum, if not the cache counter will be increased. Once it reaches quorum, the packet is forwared to the main channel for processing and the cache is reset to a negative value to avoid multiple processing.
• Each listener will react to individual errors separately and only restart.The syncing missing blocks poll will only be called when the main client reconnects and not for all clients.
• The quorum is defined as (no. of clients/2) + 1

flowchart TD
    A[Relayer] -->|Load Configs| C(Start)    
    C -->|Ws| D[Listen to events via websockets]
    C -->|Ws| E[Listen to events via websockets]
    C -->|Ws| F[Listen to events via websockets]
    D --> |event| G[internal channel]
    E --> |event| G[internal channel]
    F --> |event| G[internal channel]
    C -->|poll| Z[sync missing blocks]

    G --> H{quorum reached ??}
    H --> |No| I[add to cache \n increment counter]
    H --> |yes| J[Forward to main relayer channel\nreset counter]