🐛 Registering a data proxy sometimes clears all data proxy state

🐛 Bug Report

Sometimes when registering a data proxy the node decides it can delete the entire data-proxy configs store.

Steps to Reproduce

We initially observed the issue because it only pruned the store on 1 node which caused a chain halt. When we later executed the following transaction both nodes decided to prune the entire state.

Before

sedad query data-proxy data-proxy-config 0259c2735fec74ccdf5c49aa789080dd1de14dd0d70268d79bb4d4fc1f941cacf0
config:
  admin_address: seda123u98xsk32ra8xz3fzh9g7yu4ktr4xe7drs002
  fee:
    amount: "1000000000000000000"
    denom: aseda
  fee_update: null
  memo: ""
  payout_address: seda123u98xsk32ra8xz3fzh9g7yu4ktr4xe7drs002

After registering

sedad query data-proxy data-proxy-config 0259c2735fec74ccdf5c49aa789080dd1de14dd0d70268d79bb4d4fc1f941cacf0
Error: rpc error: code = Unknown desc = no data proxy registered for 0259c2735fec74ccdf5c49aa789080dd1de14dd0d70268d79bb4d4fc1f941cacf0: not found: unknown request

Code snippet to reproduce

sedad tx data-proxy register seda14xdjsea6w9wwfm7sgatp56pkspklgfcjwk7eus 200000000000000aseda 03f8624d228ee68a7cd1fc116a85cc43ff45a0ad07738320b32aeb3452687b2e1e 86f9528f3107972c8cd85dc03e2b1343be50ac473b37cb15e71be1fc6796fdd308ff8735ee3f96144aa927e1fa869abf59132431d85c0068cb467ba0bdd89b74 --from faucet --keyring-backend test --gas auto --gas-adjustment 1.5 --gas-prices 10000000000aseda --chain-id seda-1-devnet

Stack trace & error message

Nov 06 11:10:16 ip-172-31-114-176.eu-west-2.compute.internal cosmovisor[672370]: 11:10AM INF received complete proposal block hash=0A4C5DF6065FACEE39070EF721206A5EF7A0373313300AAF6C9594E2F6C93424 height=534798 module=consensus
Nov 06 11:10:16 ip-172-31-114-176.eu-west-2.compute.internal cosmovisor[672370]: 11:10AM INF finalizing commit of block hash=0A4C5DF6065FACEE39070EF721206A5EF7A0373313300AAF6C9594E2F6C93424 height=534798 module=consensus num_txs=2 root=E6DD752A716AAD2DE2EA4100189B9E2153DE93B3BFD86F09F0BE6658A7953C6C

Nov 06 11:10:16 ip-172-31-114-176.eu-west-2.compute.internal cosmovisor[672370]: 11:10AM ERR iavl set error error="Value missing for key [0 0 0 0 0 0 0 1 0 0 0 1] corresponding to nodeKey 73000000000000000100000001" module=server

Nov 06 11:10:16 ip-172-31-114-176.eu-west-2.compute.internal cosmovisor[672370]: 11:10AM INF finalized block block_app_hash=4A30302D478EA24F881C28EBFE793EA0C625F406535F59CC5A81139601777249 height=534798 module=state num_txs_res=2 num_val_updates=0

Expected Behavior

The behaviour should be deterministic. It would also be nice if not all data proxies get deleted for no reason. :)

Your Environment

v0.3.0-dev.3
N.A.
Linux x86_64, statically linked binary from the GH release.

sedaprotocol / seda-chain

🐛 Registering a data proxy sometimes clears all data proxy state #414