error "restoreBlocks: given chain isn't a valid continuation" when quickly creating new wallets after startup

Just checking...

[X] This is a cardano-wallet bug.
[X] I am using the latest cardano-wallet release.
[X] I am using the correct cardano-node version for that release of cardano-wallet.

Version

v2021-09-09 (git revision: fca1cdc89dbf56b328d625f7e5c539107e1d01d3)

Platform

nixos/nix:2.3.12 docker image (uname -a output: Linux 33117d699f44 5.10.47-linuxkit #1 SMP PREEMPT Sat Jul 3 21:50:16 UTC 2021 x86_64 Linux)

Installation method

Nix

Network configuration

Private testnet created using this script: https://github.com/input-output-hk/cardano-node/blob/0ed04ab8d0a205691219f04623a3d880545a3f34/scripts/byron-to-alonzo/mkfiles.sh

Context

I'm experiencing this transient cardano-wallet error in CI for about half of the CI runs:

[cardano-wallet.wallet-engine:Error:73] [2021-09-20 13:52:25.85 UTC] f58e2ec8: Unexpected error following the chain: user error (restoreBlocks: given chain isn't a valid continuation. Wallet is at: 891fcd4b<-[9c9290a4-0#0] but the given chain continues starting from: 891fcd4b<-[db619f76-14#0])

I'm running a private testnet using this script (scripts/byron-to-alonzo/mkfiles.sh alonzo && example/run/all.sh) and starting the wallet with this command (run from the same dir that mkfiles.sh was run from):

cardano-wallet serve \
    --node-socket ./example/node-bft1/node.sock \
    --testnet ./example/byron/genesis.json \
    --database ./data \
    --listen-address 0.0.0.0

Importantly, the cardano-wallet instance is started no more than 2 seconds after the node node-bft1 has created its socket file (./example/node-bft1/node.sock) and immediately after startup two new wallets are created.

Here's the full log (including that of cardano-node as well as the client speaking to cardano-wallet): https://gist.github.com/runeksvendsen/c143926249e32b22ee2d184581c9776c. The above error occurs immediately after the second wallet is created at 2021-09-20T13:52:25.8393038Z (search for "Created new wallet") in the logs here.

NB: @Anviking has suggested waiting for 3k/f slots (60 seconds using the above mkfiles.sh script) until creating the first wallet, to ensure the beginning of the chain is completely stable. This workaround has not been attempted yet.

cardano-node version: 1.29.0 (git revision: 4c5c5694d076c16b695fa0a933867509f39f2493) cardano-wallet version: v2021-09-09 (git revision: fca1cdc89dbf56b328d625f7e5c539107e1d01d3)

Description

Steps to Reproduce

First, run the cardano-node instances, and be ready to execute the below cardano-wallet "runner" script immediately after.

git clone https://github.com/input-output-hk/cardano-node.git
cd cardano-node/
scripts/byron-to-alonzo/mkfiles.sh alonzo
# run the below script now, in a separate terminal, from this directory
example/run/all.sh

cardano-wallet runner script:

SOCKET_FILE="./example/node-bft1/node.sock"

while ! ls "$SOCKET_FILE" > /dev/null ; do
    echo "waiting for cardano-node..."
    sleep 0.5
done
echo "cardano-node ready!"

cardano-wallet serve \
    --node-socket "$SOCKET_FILE" \
    --testnet ./example/byron/genesis.json \
    --database ./data &

# Generate a wallet.
# Usage: generate_wallet <wallet_name> <passphrase> <recovery_phrase>
# Returns: wallet ID
function generate_wallet() {
    local WALLET_NAME="$1"
    local PASSPHRASE="$2"
    local RECOVERY_PHRASE="$3"

    local WALLET_ID
    WALLET_ID=$(cardano-wallet wallet create from-recovery-phrase "$WALLET_NAME" < <(printf "%s\n" "$RECOVERY_PHRASE" "" "$PASSPHRASE" "$PASSPHRASE") |jq -r .id)
    echo "Created new wallet with ID $WALLET_ID" 1>&2
    echo "$WALLET_ID"
}

# generate two wallets
generate_wallet "test_wallet_1" "some very secret passphrase 1" "$(cardano-wallet recovery-phrase generate)"
generate_wallet "test_wallet_2" "some very secret passphrase 2" "$(cardano-wallet recovery-phrase generate)"

Expected behavior

cardano-wallet is able to follow the chain followed by cardano-node.

Actual behavior

cardano-wallet is not able to follow the chain followed by cardano-node.

[cardano-wallet.wallet-engine:Error:73] [2021-09-20 13:52:25.85 UTC] f58e2ec8: Unexpected error following the chain: user error (restoreBlocks: given chain isn't a valid continuation. Wallet is at: 891fcd4b<-[9c9290a4-0#0] but the given chain continues starting from: 891fcd4b<-[db619f76-14#0])

cardano-foundation / cardano-wallet