Close transaction dropped from cardano-node

[ch1bo]

Context & versions

At least 0.12.0

Steps to reproduce

1. Open a head on preprod
2. Submit the Close websocket command
3. See a transaction added to the cardano-node mempool
4. Sometimes the transaction gets removed (upon seeing the next block) without it being included.

Actual behavior

The head is not getting closed and the Cardano network just dropped our transaction for this. No user feedback is given.

Expected behavior

The Cardano network to not drop our transaction. Or at least the hydra client is made aware of this (after some time).

Hypothesis

The transaction is dropped due to the invalidAfter validity range exceeded on the closeTx.

[v0d1ch]

Some grooming notes:

• Explore if there is a way of detecting a tx dropped from a mempool (or current tx state)
  • When the txs get dropped is there any log output?
  • Is there any endpoint that we can query to know the status of a tx?
  • Is there any configuration for cardano-node that controls when items get dropped from a mempool? Mempool size? Number of elements?

Note: There is:

 `cardano-cli query tx-mempool --mainnet info/next-tx/tx-exists myTxId` 

• Potentially we could query to detect when a tx is in the mempool and when it get's dropped so we can re-send it.
• How would we test this?

[abailly-iohk]

We can observe the Mempool using a specialised mini-protocol we could implement of client for, but this is somewhat involved. We could do something simpler using timeouts in the HeadLogic: When you request a Close, use a Wait to have an upper bound on how long you're waiting for observing the OnCloseTx from the chain?

[ch1bo]

When you request a Close, use a Wait to have an upper bound on how long you're waiting for observing the OnCloseTx from the chain?

You mean some retrying logic in the HeadLogic?

---

I think we should just adjust the upper validity range to be something more compatible with the network. This is the code which determines that upper bound: https://github.com/input-output-hk/hydra/blob/11dc6aeb68b909e070fc6eb366b4734e090b62c1/hydra-node/src/Hydra/Chain/Direct/Handlers.hs#L354-L358 Looking at that, I wonder whether this is really the issue we encounter? 200 seconds is long enough for mainnet. But then again, the contestation period is configurable and the default of 60 seconds might not be long enough (if a block is not produced within 1 minute?) https://github.com/input-output-hk/hydra/blob/11dc6aeb68b909e070fc6eb366b4734e090b62c1/hydra-node/src/Hydra/Options.hs#L605-L606

We should validate whether it's due to upper validity bounds first.

[abailly-iohk]

It would be great to keep track of the block hash/height when this happens in order to validate the invalid upper bound hypothesis. In the meantime, I would like to suggest we address this issue by documenting this possible behaviour and let client applications (eg. hydra-tui or any frontend apps interacting with a hydra-node) decide what to do based on their perception of time.

[ch1bo]

I think this appeared in this smoke test: https://github.com/input-output-hk/hydra/actions/runs/7088575057

[abailly]

Happened to me today while closing a head with team: Tx got successfully posted to the cardano-node but it was later on removed from mempool. There's no trace in the node telling why this happened.