Bug: Extrinsic is accepted by the node but not included into the block

[ERussel]

Description

Batch extrinsic that includes omnipool swap and a call to revert fee to HDX with the ref_time estimated 6_167_235_546 is accepted by the node but not included into the block. Finally extrinsic expires because of the mortality is set (5 minutes). However the bug is floating and looks like it depends on network conditions. However we had high chances to catch it during last month.

Expected Behavior

Extrinsic doesn't look heavy taken into account that max extrinsic ref_time for normal class is 362_375_586_000. So the expected behaviour is that the node accepts the extrinsic and includes it into the block within the mortality given (5 minutes).

Actual Behavior

Extrinsic is not included into the block and getting expired due to mortality (5 minutes)

Possible Fix

We see that Hydra has quite high mandatory weight in the block. Almost 99% of max_block. That might be a reason. But we notice that the blockchain sometimes adapts and somehow reduces the mandatory weight to include even bigger extrinsics that one we have.

Steps to Reproduce

Extrinsic example

``` [ { isSigned: true method: { args: { calls: [ { args: { asset_in: 9 asset_out: 0 amount: 501,864,492,697,392,013 min_buy_amount: 5,300,410,956,267 } method: sell section: omnipool } { args: { currency: 0 } method: setCurrency section: multiTransactionPayment } ] } method: forceBatch section: utility } assetId: null era: { MortalEra: { period: 64 phase: 57 } } metadataHash: null mode: 1 nonce: 9 signature: 0xd2508de41104b5a5a0a45e9797a2bec0f68852e92a4205fc930c43cc78f1987812e69c89b5c0a12f7f02e348324ad0efcc09b0ee2043f60751cc51dbc39c2989 signer: 7LWL7xdzgiCercrh1EgaiHnBE5orofv6rnuAUv54XRTYqCdk tip: 0 } ] ```

[mrq1911]

i was not able to replicate that issue, are you sure you are setting correct nonce in the extrinsic? setting nonce too high could cause exactly this

make sure you are not incrementing nonces for canceled (not broadcasted) extrinsic

[valentunn]

Hey @mrq1911 here is an example that shows that nonce is not the issue. (accountNextIndex is 1 because of the pending tx)