RPC: pending extrinsics entering txpool

[zees-dev]

Previously raised here: https://github.com/paritytech/polkadot-sdk/issues/3473

Motivation

Was looking for a subscription (websocket pubsub) to monitor the live txpool/mempool for incoming extrinsics. There is the author_pendingExtrinsics RPC; but this gives a snapshot view of the txpool upon request.

Request

A new RPC endpoint which should simply allow a caller to subscribe to all incoming pending extrinsics.

Once a user subscribes, the RPC endpoint will stream out live incoming transactions (type Extrinsic) in the Ready state. The transactions do not need to be finalized - as that defeats the purpose since the state has already been set.

Solution

A new subscription in the client/rpc-api/src/author/mod.rs file. Something like so:

    /// Watch transaction txpool.
    #[subscription(
        name = "author_watchTxpool" => "author_txpoolUpdate",
        unsubscribe = "author_unwatchTxpool",
        item = sp_runtime::OpaqueExtrinsic, // Transaction (scale-encoded-extrinsic)
    )]
    fn watch_txpool(&self);

This should be fairly simple to implement (already have a local implementation building/streaming out to WS connection).

Example response (scale encoded extrinsic):

{"jsonrpc":"2.0","method":"author_txpoolUpdate","params":{"subscription":"GcJD5bwBSSzLMNIK","result":"0x8d018425451a4de12dccc2d166922fa938e900fcc4ed244f7826dbd9e7984784cd33ea3c23f24509fa4f77bf56c509692c4d1e20c7541d62677cf055d32e078d2b38a30cd2ab244cbde87b3ab88fcc8997da043e770a760186010031110001147375702032"}}

Potential concerns

If there are issues (DoS, etc.); maybe this could be put behind feature flags - similar to what's done in frontier for the ethereum txpool subscriptions.

Are you willing to help with this request?

Yes!

[tomaka]

What's your use case for such a feature?

monitor the live txpool/mempool for incoming extrinsics.

How do you define "incoming extrinsics"? Do you also include in that the transactions that were received from other nodes on the peer-to-peer network?

[zees-dev]

o you also include in that the transactions that were received from other nodes on the peer-to-peer network?

Yes this is correct; transactions which are in the ready state (ready pool), but not yet finalized to be included in the block. The usecase is to be able to get the live, un-finalized state for transactions and figure out where extrinsics of interest lie within the queue; this could be used for monitoring purposes, displaying info on an explorer, bot-making/MEV, etc.

[tomaka]

This would need to be a sudo-prefixed function, as each node has a different "opinion" on what this of transactions is. It wouldn't be implementable by light clients.

displaying info on an explorer

This information really shouldn't be displayed in an explorer. The transactions in the queue might not actually be valid, plus you will most likely only have a partial view of the list of all transactions that are waiting to be included.

If you're thinking of work-arounds for these two issues, for example querying multiple nodes, is kind of counter-productive. The entire reason why blockchains exist in the first place is precisely to solve these two issues.

[zees-dev]

Im assuming the same issues exist in geth (not to say this is an architecture one should look upto); and yes teams/searchers participating in MEV do subscribe to the mempool of multiple nodes to get an accurate picture of the live state of the mempool.

Another use-case: If you for example wanted to create bots which stabilized the price after an AMM swap - to provide best price for users, or building some other service which acts on the unfinalized data (which is normal in other chains) - then currently this isn't possible for substrate based chains (out of the box atleast).

The current pendingExtrinsics RPC also gives the same view - of a single node's ready queue; this subscription would just add more value to that.

WRT to MEV, this RPC just levels the playing field a bit since advanced players can make code changes to their nodes to achieve this functionality (which I'm planning to do); thought it would be a better option if this just came out of the box with the substrate based chains.

[AndreasGassmann]

I would also be interested in this feature. The use case is to provide a real-time and interactive view of what's going on on-chain. In the UI, you can display "pending" extrinsics, and then mark them as included once they are in a block, for example. You can already do that by querying the current API frequently, but having a subscribe API would be nicer.

Some example use-cases:

https://txcity.io/v/eth-btc - A visualization of txs as people and blocks as cars. https://blocks.wizb.it/ - A world view of where txs are being injected (probably highly inaccurate, but doesn't matter) https://tzflow.com/ - A website showing the network activity of Tezos. Pending txs, and then you can see if they were included, failed or skipped.