/v2/proxy/transactions endpoint extremely slow when network is congested

[refi93]

Just checking...

• [X] This is a cardano-wallet bug.
• [X] I am using the latest cardano-wallet release.
• [X] I am using the correct cardano-node version for that release of cardano-wallet.

Version

inputoutput/cardano-node:1.30.1

Platform

Ubuntu 20.04 (should not be relevant)

Installation method

IOHK Docker image

Network configuration

mainnet

Context

Around the time the network is congested, we noticed that transactions may take a very long time to be accepted by the node (> 15 minutes) which means the transaction submission request takes so long that it times out, leaving the transaction in kind of a "limbo" and from the app perspective it may even look as a failure (a 5xx error - most likely 504 - gateway timeout), if the request is proxied

Attaching examples from logs documenting the issue:

^[[34m[cardano-wallet.wallet-engine:Info:3496]^[[0m [2021-10-04 19:24:20.54 UTC] Transaction d26210f8 accepted by local node
^[[31m[cardano-wallet.api-server:Error:3496]^[[0m [2021-10-04 19:24:20.54 UTC] [RequestId 911] POST /v2/proxy/transactions 202 Accepted in 1051.299354215s

^[[31m[cardano-wallet.api-server:Error:3491]^[[0m [2021-10-04 19:23:09.66 UTC] [RequestId 907] POST /v2/proxy/transactions 202 Accepted in 984.213215602s
^[[34m[cardano-wallet.wallet-engine:Info:3491]^[[0m [2021-10-04 19:23:09.66 UTC] Transaction 525cb830 accepted by local node

Description

Around the time the network is congested, we noticed that transactions may take a very long time to be accepted by the node (> 15 minutes) which means the transaction submission request takes so long that it times out, leaving the transaction in kind of a "limbo"

Reproduction - hard to reproduce, apparently related to the cardano network sometimes being congested with too many transactions

Not sure what is a proper solution to this problem, probably related to https://github.com/input-output-hk/cardano-node/issues/3247 , but cardano-wallet should probably have a way to submit a transaction without awaiting its acceptance into the blockchain so the submitting party can get the acknowledgement that the submission itself went through (understanding that the transaction may or may not eventually make it to the blockchain)

[piotr-iohk]

@paweljakubas @KtorZ is this something that we could potentially address with tx re-submission mechanism (presumably added to new submit tx endpoint https://github.com/input-output-hk/cardano-wallet/pull/2959)?

I understand that POST /v2/proxy/transactions is mostly used with HW wallets integration. It has no re-submission. I suppose that new tx submit ep could be potentially used in the context of HW wallets as well. :thinking:

[KtorZ]

should probably have a way to submit a transaction without awaiting its acceptance into the blockchain

This is actually already how the submission operates. Right now, the protocol only awaits for the transaction to be accepted in the node's local mempool. Yet, if the mempool is full, the protocol is blocking until there's room for the transaction as a mechanism to apply back pressure on clients.

While we would imagine an extra buffer in the wallet to accept transactions and automatically try to submit them when the mempool has enough space, It's only really shifting the problem from the client to the wallet backend. In the end, those transactions won't be included faster by doing this and, blocking on the submission is actually a proper way to handle this.

In brief, I don't think there's really anything that the wallet can do here for it is constrained by the upstream limitations.

[refi93]

I see, at the same time if the request takes that long and it's made from a browser or via another service such as heroku, there may be globally imposed timeouts which are well below these let's say 10-15 minutes a tx submission request can take. Does the transaction submission from cardano-wallet's perspective continue if the request is aborted by the caller?