Exchange breaks with constant `CustomError` after commit 29c2a63c

[jackcmay]

Problem

After commit: * 29c2a63c - Retry transactions that failed due to account lock (#4184) (23 hours ago) <Pankaj Garg>

The exchange fails before trading while its attempting to create token accounts. If that specific commit is reverted the issue does not present itself.

The following is an example of the error that produced repeatidly once the behavior starts:

[2019-05-08T17:07:18.365753000Z ERROR solana_runtime::bank] tx error: Err(InstructionError(0, CustomError(0))) Transaction { signatures: [5hPC9aUSmAUaZycsWDZtGPkpG5MswvD1QhXMMFzTF8DjrdrWET1SMjd7HfPetHgHqwNfmH3b4DmMWeghcdB6tLUt], message: Message { num_required_signatures: 1, account_keys: [7mYJe5WoHKyJaPEzTRtbr8Va5aPR9VZMHxkaUxAzFeDa, CaByj1mhq1s2qFMh71CT4BfmLGE5yVxfowQEqBYE7CP2], recent_blockhash: 8Q3bxpesweQ19fc1C4tA5UDR2GsuGUXkAbKKBguY86GX, program_ids: [11111111111111111111111111111111, Exchange11111111111111111111111111111111111], instructions: [CompiledInstruction { program_ids_index: 0, accounts: [0, 1], data: [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 72, 0, 0, 0, 0, 0, 0, 0, 3, 147, 111, 103, 210, 47, 14, 213, 108, 116, 49, 115, 232, 171, 14, 111, 167, 140, 221, 234, 33, 70, 185, 192, 42, 31, 141, 152, 0, 0, 0, 0] }, CompiledInstruction { program_ids_index: 1, accounts: [0, 1], data: [0, 0, 0, 0] }] } }
[2019-05-08T17:07:18.365808000Z INFO  solana_runtime::bank] 2 errors of 2 txs

To recreate run the exchange local cluster test:

Change the test_exchange_local_cluster's config (to create more token acounts) on line 928 to:

        config.duration = Duration::from_secs(5000);
        config.fund_amount = 100_000;
        config.threads = 1;
        config.transfer_delay = 150; // 15
        config.batch_size = 1000; // 1000;
        config.chunk_size = 100; // 200;
        config.account_groups = 10; // 10;

Run the exchange test:

$ cd bench-exchange
$ RUST_LOG=solana_bench_exchange=info,solana_exchange_api=info,solana=info cargo test --release -- --nocapture test_exchange_local_cluster

Proposed Solution

[jackcmay]

Looks like @sakridge is also seeing this issue

[pgarg66]

https://github.com/solana-labs/solana/pull/4214 https://github.com/solana-labs/solana/pull/4199