[Bug] Inconsistency in transaction rejection and client response.

[CindyZhouYH]

🐛 Bug: Inconsistency in transaction rejection and client response.

I built a 4-node chain locally following the document and used the python sdk to interact with it. When I'm sending transactions concurrently, some transactions that should have triggered MempoolStatusCode::InvalidUpdate and thrown an 'Send Transaction Exception' are instead acting as submitted successfully, returning Response [202 Accepted] and a transaction hash. However, it seems that these transactions are not actually submitted.

To reproduce

Code snippet to reproduce The python sdk script:

# the rest APIs of the chain
nodes = ["http://127.0.0.1:xxx", "http://127.0.0.1:xxx", "http://127.0.0.1:xxx", "http://127.0.0.1:xxx"]
async def create_accounts(num):
    for i in range(0, num):
        new_acc = Account.generate()
        new_acc.store(account_file)
        new_fund = faucet_client.fund_account(new_acc.address(), 100_000_000)
        accounts.append(new_acc)
        fund.append(new_fund)
    await asyncio.gather(*fund)
    return accounts

async def parallel_vector(target_address: AccountAddress, account_num: int, op_num: int):
    print("Start creating accounts")
    accounts = await create_accounts(account_num)
    print("Account create finished!")

    # init multiple clients
    for node in nodes:
        clients.append(RestClient(node + "/v1"))

    print("Start generating transactions")
    for i in range(0, op_num):
        print(i)
        account = random.choice(accounts)
        client = random.choice(clients)
        payload = EntryFunction.natural(
                f"{target_address}:: ModuleOne",
                "add_vector",
                [],
                [TransactionArgument(target_address, Serializer.struct), TransactionArgument(i, Serializer.u64)],
            )
        try:
            signed_transaction = await client.create_bcs_signed_transaction(
                account, TransactionPayload(payload)
            )
            txn_hash = await client.submit_bcs_transaction(signed_transaction)
            transactions.append(txn_hash)
            print("Transaction {} Submitted".format(txn_hash))
        except Exception as e:
            print("Send Transaction Exception: " + str(e))
            continue
    print("All {} Transactions Submitted".format(len(transactions)))
    print("All transaction hashes: ")
    print(transactions)
    for tx in transactions:
        await random.choice(clients).wait_for_transaction(tx)

if __name__ == "__main__":
    account_address = AccountAddress.from_str(deployed_addr)
    asyncio.run(parallel_vector(account_address, 2, 5))

The move module:

module test_module::ModuleOne {
    use std::error;
    use std::signer;
    use std::vector;

    struct Vec has key {
        vec: vector<u64>
    }

    public entry fun enter_initialize(account: signer) {
        let account_addr = signer::address_of(&account);
        if (!exists<Vec>(account_addr)) {
            move_to(&account, Vec { vec: vector::empty<u64>() });
        };
    }

    public entry fun add_vector(addr: address, num: u64) acquires Vec {
        let vec = &mut borrow_global_mut<Vec>(addr).vec;
        vector::push_back(vec, num);
    }

Stack trace/error message The output:

However, two of the Response [202 Accepted] transactions are not submitted and cannot be found.

root@xxx:~# curl --request GET   --url http://127.0.0.1:39991/v1/transactions/by_hash/0x9d55aee3edc67ad8376d1f5ce15ee1bc4383c0195b70e78666099b2359369f2a
{"message":"Transaction not found by Transaction hash(0x9d55aee3edc67ad8376d1f5ce15ee1bc4383c0195b70e78666099b2359369f2a)","error_code":"transaction_not_found","vm_error_code":null}
root@xxx:~# curl --request GET   --url http://127.0.0.1:39991/v1/transactions/by_hash/0xe6cce0422bfa08776883d49cd40e3f2558634415242e46dedc55b7ac36f13dc4
{"message":"Transaction not found by Transaction hash(0xe6cce0422bfa08776883d49cd40e3f2558634415242e46dedc55b7ac36f13dc4)","error_code":"transaction_not_found","vm_error_code":null}

Expected Behavior

The rejected transactions should through the Send Transaction Exception. The transactions with Response [202 Accepted] and a returned transaction hash should be found on the chain.

System information

Please complete the following information:

• Aptos Core Version: Aptos Node 1.8.0
• Rust Version: 1.74.1
• Computer OS: Ubuntu 22.04

[gregnazario]

Transactions sent to the same node, will always be found, transactions sent to different nodes may not (if they're not accepted).

The flow works as follows:

1. You submit to node A (you get a 202 that it was accepted to the Mempool)
2. Node B, C, D all get a propagated version of it, and submit it to consensus
3. If it's committed (succeed or fail), it will show up in the other nodes

Now there are a few caveats where the transactions you submitted won't show up. This is if the Mempool doesn't propagate it, because it knows that it will not suceed:

1. If transaction expired, it may be dropped
2. If the transaction's sequence number has already been used it may be dropped prior to committing on-chain
3. If the transaction spends more than X minutes in the Mempool.

@bchocho can provide more details here

[gregnazario]

It can get really confusing because if I say submit 4 different transactions with sequence number 1 to 4 different nodes. Only one will win, and the other 3 may or may not be committed with a failed transaction.