txs_results not returned properly on dYdX nodes

[penso]

Bug Report

Setup

dYdX node, currently using v0.38.5 (code)

What happened?

Fetching older block_results doesn't return txs_results as it should. This block/block_results was saved in November and properly has 4 txs and 4 txs_results (this JSON combines both block and block_results)

That same block_results fetched today at https://dydx-ops-archive-rpc.kingnodes.com/block_results?height=10000 only has 1 txs_results, and missing 3 txs_results. I feel like txs_results created with v0.37.2 aren't returned properly when upgraded to v0.38. I also tried a few archive nodes and they all behave the same way.

I'm not affiliated with dYdX, I just happen to try running my cosmos indexer (Constellations) on dYdX and noticed it failed because the amount of txs_results is expected to be the same as txs.

[penso]

Someone on a channel said there is the same issue on sei devnet

[andynog]

we are currently investigating the problem and we will add more context here

[andynog]

I could reproduce the problem only on CometBFT side. There is a way to reproduce it using only the kvstore example app.

Here are the steps:

First you need to run a cometbft v0.37. On a terminal in a cometbft repo folder, run:

git checkout v0.37.x
make install
cometbft init --home /tmp/cometbft-kvstore
cometbft start --proxy_app=kvstore --home=/tmp/cometbft-kvstore

Then submit a tx:

curl --location 'localhost:26657/broadcast_tx_commit?tx=%22cometbft%3Drocks%22'

Which will output a JSON

{"jsonrpc":"2.0","id":-1,"result":{"check_tx":{"code":0,"data":null,"log":"","info":"","gas_wanted":"1","gas_used":"0","events":[],"codespace":"","sender":"","priority":"0","mempoolError":""},"deliver_tx":{"code":0,"data":null,"log":"","info":"","gas_wanted":"0","gas_used":"0","events":[{"type":"app","attributes":[{"key":"creator","value":"Cosmoshi Netowoko","index":true},{"key":"key","value":"cometbft","index":true},{"key":"index_key","value":"index is working","index":true},{"key":"noindex_key","value":"index is working","index":false}]},{"type":"app","attributes":[{"key":"creator","value":"Cosmoshi","index":true},{"key":"key","value":"rocks","index":true},{"key":"index_key","value":"index is working","index":true},{"key":"noindex_key","value":"index is working","index":false}]}],"codespace":""},"hash":"1B3C5A1093DB952C331B1749A21DCCBB0F6C7F4E0055CD04D16346472FC60EC6","height":"60"}}%   

Make note of the height in the end of the JSON returned. So in this case it was height 60. Query the block_results for that height:

curl --location 'localhost:26657/block_results?height=60' | jq .

Which will return

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "height": "60",
    "txs_results": [
      {
        "code": 0,
        "data": null,
        "log": "",
        "info": "",
        "gas_wanted": "0",
        "gas_used": "0",
        "events": [
          {
            "type": "app",
            "attributes": [
              {
                "key": "creator",
                "value": "Cosmoshi Netowoko",
                "index": true
              },
              {
                "key": "key",
                "value": "cometbft",
                "index": true
              },
              {
                "key": "index_key",
                "value": "index is working",
                "index": true
              },
              {
                "key": "noindex_key",
                "value": "index is working",
                "index": false
              }
            ]
          },
          {
            "type": "app",
            "attributes": [
              {
                "key": "creator",
                "value": "Cosmoshi",
                "index": true
              },
              {
                "key": "key",
                "value": "rocks",
                "index": true
              },
              {
                "key": "index_key",
                "value": "index is working",
                "index": true
              },
              {
                "key": "noindex_key",
                "value": "index is working",
                "index": false
              }
            ]
          }
        ],
        "codespace": ""
      }
    ],
    "begin_block_events": null,
    "end_block_events": null,
    "validator_updates": null,
    "consensus_param_updates": null
  }
}

Note that in the txs_results there's a value returned with events

Stop the running cometbft with ctrl+c

Now let's run an instance of cometbft v0.38. Please note that this time we are just replacing cometbft binary and starting it again, still keeping the same config and data folder (cometbft home) as before:

git checkout v0.38.x
make install
cometbft start --proxy_app=kvstore --home=/tmp/cometbft-kvstore

Now let's query the same height as before (60):

curl --location 'localhost:26657/block_results?height=60' | jq .

and its return

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "height": "60",
    "txs_results": [
      {
        "code": 0,
        "data": null,
        "log": "",
        "info": "",
        "gas_wanted": "0",
        "gas_used": "0",
        "events": [],
        "codespace": ""
      }
    ],
    "finalize_block_events": [
      {}
    ],
    "validator_updates": [
      {
        "pub_key": {
          "Sum": null
        }
      }
    ],
    "consensus_param_updates": null,
    "app_hash": null
  }
}

Note the txs_results doesn't return properly, event though there's one result, it's all empty (no events).

Now, just for the sake of testing, let's submit another transaction in v0.38

curl --location 'localhost:26657/broadcast_tx_commit?tx=%22cometbft%3Drocks%22'

You should get again a JSON response with a height. Use that height to query the block_results. In my test I've got height 395

curl --location 'localhost:26657/block_results?height=395' | jq .               

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "height": "395",
    "txs_results": [
      {
        "code": 0,
        "data": null,
        "log": "",
        "info": "",
        "gas_wanted": "0",
        "gas_used": "0",
        "events": [
          {
            "type": "app",
            "attributes": [
              {
                "key": "creator",
                "value": "Cosmoshi Netowoko",
                "index": true
              },
              {
                "key": "key",
                "value": "cometbft",
                "index": true
              },
              {
                "key": "index_key",
                "value": "index is working",
                "index": true
              },
              {
                "key": "noindex_key",
                "value": "index is working",
                "index": false
              }
            ]
          },
          {
            "type": "app",
            "attributes": [
              {
                "key": "creator",
                "value": "Cosmoshi",
                "index": true
              },
              {
                "key": "key",
                "value": "rocks",
                "index": true
              },
              {
                "key": "index_key",
                "value": "index is working",
                "index": true
              },
              {
                "key": "noindex_key",
                "value": "index is working",
                "index": false
              }
            ]
          }
        ],
        "codespace": ""
      }
    ],
    "finalize_block_events": null,
    "validator_updates": null,
    "consensus_param_updates": null,
    "app_hash": null
  }
}

Note that the tx submitted and queried with v0.38 returns properly (with events).

So this confirms the problem exists between releases from cometbft. Another test I've performed was to revert cometbft to v0.37 and query that block_result that was not returned properly in v0.38 and in v0.37 it returned properly. So this is not an issue with storage.

I've done additional troubleshooting and the problem is related on how the ABCI response is retrieved from the store to be used in the RPC block_results endpoint. The logic has to handle legacy ABCI responses and currently is not doing it properly.

I'm already testing a fix and will follow up later with that.