celestiaorg / celestia-app

Celestia consensus node
https://celestiaorg.github.io/celestia-app/
Apache License 2.0
335 stars 275 forks source link

PFB results in panic: UnmarshalJSON cannot decode empty bytes #1363

Closed rootulp closed 1 year ago

rootulp commented 1 year ago

Summary of Bug

recovered: UnmarshalJSON cannot decode empty bytes
stack:
goroutine 3717 [running]:
runtime/debug.Stack()   /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x1f63960, 0xc0065ba800})  /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:71 +0x27
github.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x1f63960?, 0xc0065ba800?})
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:39 +0x30
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x1f63960, 0xc0065ba800}, 0xc001344b00?)
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:28 +0x37
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x1f63960, 0xc0065ba800}, 0x2d32158?)
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/recovery.go:33 +0x5e
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/baseapp.go:639 +0xf0
panic({0x1f63960, 0xc0065ba800})
    /usr/local/go/src/runtime/panic.go:838 +0x207
github.com/cosmos/cosmos-sdk/x/params/types.Subspace.Get({{0x2d31748, 0xc000f62bf0}, 0xc000122bd0, {0x2d0e170, 0xc000f62e50}, {0x2d0e1c0, 0xc000f62ef0}, {0xc000052600, 0x4, 0x12}, ...}, ...)
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/x/params/types/subspace.go:110 +0x307
github.com/celestiaorg/celestia-app/x/blob/keeper.Keeper.GasPerBlobByte(...)
    /celestia-app/x/blob/keeper/params.go:36
github.com/celestiaorg/celestia-app/x/blob/keeper.Keeper.PayForBlobs({{0x2d31748, 0xc000f62bf0}, {0x2d0e170, 0xc000f62ea0}, {0x2d0e170, 0x0}, {{0x2d31748, 0xc000f62bf0}, 0xc000122bd0, {0x2d0e170, ...}, ...}}, ...)
    /celestia-app/x/blob/keeper/keeper.go:60 +0x3a8
github.com/celestiaorg/celestia-app/x/blob.NewHandler.func1({{0x2d24c58, 0xc000138020}, {0x2d32158, 0xc006a22680}, {{0xb, 0x0}, {0xc006eec050, 0x9}, 0x19ed, {0xb752ea5, ...}, ...}, ...}, ...)
    /celestia-app/x/blob/handler.go:20 +0x195
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x2d24c58, 0xc000138020}, {0x2d32158, 0xc006a22680}, {{0xb, 0x0}, {0xc006eec050, 0x9}, 0x19ed, ...}, ...}, ...)
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/baseapp.go:800 +0x7a8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc000facfc0, 0x3, {0xc000540f20, 0x141, 0x141})
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/baseapp.go:725 +0xc45
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc000facfc0, {{0xc000540f20?, 0x20?, 0xc00dc7a460?}})
    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.8.0-sdk-0.46.7/baseapp/abci.go:283 +0x17a
github.com/tendermint/tendermint/abci/client.(*localClient).DeliverTxAsync(0xc0012e7740, {{0xc000540f20?, 0x0?, 0xc0012e7740?}})
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/abci/client/local_client.go:93 +0x105
github.com/tendermint/tendermint/proxy.(*appConnConsensus).DeliverTxAsync(0xc00def4bc0?, {{0xc000540f20?, 0x20?, 0xb?}})
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/proxy/app_conn.go:88 +0x26
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2d25e10?, 0xc003416cc0}, {0x2d2ffe8, 0xc0011e3b20}, 0xc00163e000, {0x2d329b0, 0xc000130978}, 0x19ec?)
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/state/execution.go:390 +0x822
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0033f33d0, 0x7}}, {0xc0033f33d7, 0x9}, 0x1, 0x19ec, {{0xc00c1e5cc0, ...}, ...}, ...}, ...)
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/state/execution.go:210 +0x171
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc00136aa80, 0x19ed)
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:1668 +0xb2d
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc00136aa80, 0x19ed)
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:1576 +0x2ff
github.com/tendermint/tendermint/consensus.(*State).handleCompleteProposal(0xc00136aa80, 0x454?)
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:1957 +0x399
github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00136aa80, {{0x2d06580?, 0xc003c70990?}, {0xc002e3f1d0?, 0x0?}})
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:834 +0x1c5
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc00136aa80, 0x0)
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:760 +0x419
created by github.com/tendermint/tendermint/consensus.(*State).OnStart
    /go/pkg/mod/github.com/celestiaorg/celestia-core@v1.14.0-tm-v0.34.23/consensus/state.go:379 +0x12d
: panic

Originally reported by @renaynay

Version

https://github.com/celestiaorg/celestia-app/releases/tag/v0.12.0-rc6

Steps to Reproduce

Submit a PFB on Arabica with

{"namespace_id": "0c204d39600fddd3", "data": "f1f20ca8007e910a3bf8b2e61da0f26bca07ef78717a6ea54165f5", "gas_limit": 700000, "fee": 2000}
rootulp commented 1 year ago

We currently suspect that the GasPerBlobByte param was not defined in the genesis file for the current iteration of the Arabica testnet.

evan-forbes commented 1 year ago

if we just didn't set the param, then the safest quickest thing in this case (single validator node on arabica), would be to hardfork and change the param manually via an upgrade handler esque script

either that, or we submit a proposal to update it via gov. iirc, we should be able to push a vote through that chain very quickly

rootulp commented 1 year ago

@Bidon15 confirmed that when Arabica was upgraded the param was not.

via an upgrade handler esque script

Are you referring to an x/upgrade plan?

or we submit a proposal to update it via gov

Seems possible via CLI commands draft proposal and submit proposal. I'll look into preparing a proposal.

What are the downsides to creating a new genesis file with the param populated and starting arabica-5 / shutting down arabica-4?

evan-forbes commented 1 year ago

Are you referring to an x/upgrade plan?

we can hack it by just including a script in the binary then switch the only validator with that binary.

What are the downsides to creating a new genesis file with the param populated and starting arabica-5 / shutting down arabica-4?

that's also a very good option!!

rootulp commented 1 year ago

@Bidon15 @sysrex how feasible is it to start a new chain with an updated genesis file? The field that needs to be added to genesis.json is:

    "blob": {
      "params": {
        "min_square_size": 1,
        "max_square_size": 128,
+       "gas_per_blob_byte": 8
      }
    },
rootulp commented 1 year ago

Options:

  1. Stop arabica-4. Start arabica-5 with a genesis.json file that includes the param above.
  2. Draft and submit a governance proposal that sets the param on arabica-4. From the validator's account, vote yes on the proposal. Wait until it goes into effect.
  3. Write a manual script in the celestia-app repo that sets the param. Get the script reviewed, merged, and included in the celestia-appd binary. Cut a new release candidate (potentially v0.12.0-rc8). Upgrade the validator on arabica-4 to the new release candidate.
rootulp commented 1 year ago

@sysrex we explored option 2 with a legacy governance proposal:

{
   "title":"Parameter change: initialize gas_per_blob_byte",
   "description":" set the gas_per_blob_byte param to 8",
   "changes":[
      {
         "subspace":"blob",
         "key":"gas_per_blob_byte",
         "value":"8"
      }
   ],
   "deposit":"1000utia"
}

We submitted the governance proposal but it failed because we're trying to initialize a parameter that hasn't been registered yet. Ref:

  1. https://github.com/cosmos/cosmos-sdk/blob/8bd9288051d464a93d1101b189d7cb00c53beef1/x/params/types/subspace.go#L204-L214
  2. https://github.com/cosmos/cosmos-sdk/blob/8bd9288051d464a93d1101b189d7cb00c53beef1/x/params/proposal_handler.go#L37

So we're going to try option 1 now.

rootulp commented 1 year ago

We expect this to be resolved by deployment of arabica-5 which is scheduled for tomorrow morning.

rootulp commented 1 year ago

Submitting PFBs on arabica-5 appears to work.

$ curl -X POST -d '{"namespace_id": "0c204d39600fddd3",
  "data": "f1f20ca8007e910a3bf8b2e61da0f26bca07ef78717a6ea54165f5",
  "gas_limit": 140000, "fee": 2000}' http://localhost:26659/submit_pfb

Output:

{
  "height": 10573,
  "txhash": "E2CD12457D9263C98629B018C3F7C7D68B779FC90EA539462B94EEBB6D7191B3",
  "data": "122A0A282F63656C65737469612E626C6F622E76312E4D7367506179466F72426C6F6273526573706F6E7365",
  "raw_log": "[{\"msg_index\":0,\"events\":[{\"type\":\"celestia.blob.v1.EventPayForBlobs\",\"attributes\":[{\"key\":\"blob_size\",\"value\":\"1\"},{\"key\":\"namespace_ids\",\"value\":\"[\\\"DCBNOWAP3dM=\\\"]\"},{\"key\":\"signer\",\"value\":\"\\\"celestia1vdjkcetnw35kzvtv89ckvepcdpcx6mr2xd4nwur8dp6hydryxph8xdtyxdsnsumrwaekxumtwaksaphpv9\\\"\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"/celestia.blob.v1.MsgPayForBlobs\"}]}]}]",
  "logs": [
    {
      "msg_index": 0,
      "events": [
        {
          "type": "celestia.blob.v1.EventPayForBlobs",
          "attributes": [
            {
              "key": "blob_size",
              "value": "1"
            },
            {
              "key": "namespace_ids",
              "value": "[\"DCBNOWAP3dM=\"]"
            },
            {
              "key": "signer",
              "value": "\"celestia1vdjkcetnw35kzvtv89ckvepcdpcx6mr2xd4nwur8dp6hydryxph8xdtyxdsnsumrwaekxumtwaksaphpv9\""
            }
          ]
        },
        {
          "type": "message",
          "attributes": [
            {
              "key": "action",
              "value": "/celestia.blob.v1.MsgPayForBlobs"
            }
          ]
        }
      ]
    }
  ],
  "gas_wanted": 140000,
  "gas_used": 76252,
  "events": [
    {
      "type": "coin_spent",
      "attributes": [
        {
          "key": "c3BlbmRlcg==",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        },
        {
          "key": "YW1vdW50",
          "value": "MjAwMHV0aWE=",
          "index": true
        }
      ]
    },
    {
      "type": "coin_received",
      "attributes": [
        {
          "key": "cmVjZWl2ZXI=",
          "value": "Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=",
          "index": true
        },
        {
          "key": "YW1vdW50",
          "value": "MjAwMHV0aWE=",
          "index": true
        }
      ]
    },
    {
      "type": "transfer",
      "attributes": [
        {
          "key": "cmVjaXBpZW50",
          "value": "Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=",
          "index": true
        },
        {
          "key": "c2VuZGVy",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        },
        {
          "key": "YW1vdW50",
          "value": "MjAwMHV0aWE=",
          "index": true
        }
      ]
    },
    {
      "type": "message",
      "attributes": [
        {
          "key": "c2VuZGVy",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        }
      ]
    },
    {
      "type": "tx",
      "attributes": [
        {
          "key": "ZmVl",
          "value": "MjAwMHV0aWE=",
          "index": true
        },
        {
          "key": "ZmVlX3BheWVy",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20=",
          "index": true
        }
      ]
    },
    {
      "type": "tx",
      "attributes": [
        {
          "key": "YWNjX3NlcQ==",
          "value": "Y2VsZXN0aWExbDlxZmQ4aHBtbGozazdwZ2h1cjRkMG5zNWQzYThzY3dzY3Nrd20vMA==",
          "index": true
        }
      ]
    },
    {
      "type": "tx",
      "attributes": [
        {
          "key": "c2lnbmF0dXJl",
          "value": "eml2QnpZSy9mUWEwbklLUURBYzF2djVKZFNQUEFlK1RJQlJDVTd5dElCd2xpUGxRaVJLNFhKZUVoNXBxdWtaU3JTOHlsOVN4UUZXTWM0b293SFkySEE9PQ==",
          "index": true
        }
      ]
    },
    {
      "type": "message",
      "attributes": [
        {
          "key": "YWN0aW9u",
          "value": "L2NlbGVzdGlhLmJsb2IudjEuTXNnUGF5Rm9yQmxvYnM=",
          "index": true
        }
      ]
    },
    {
      "type": "celestia.blob.v1.EventPayForBlobs",
      "attributes": [
        {
          "key": "YmxvYl9zaXpl",
          "value": "MQ==",
          "index": true
        },
        {
          "key": "bmFtZXNwYWNlX2lkcw==",
          "value": "WyJEQ0JOT1dBUDNkTT0iXQ==",
          "index": true
        },
        {
          "key": "c2lnbmVy",
          "value": "ImNlbGVzdGlhMXZkamtjZXRudzM1a3p2dHY4OWNrdmVwY2RwY3g2bXIyeGQ0bnd1cjhkcDZoeWRyeXhwaDh4ZHR5eGRzbnN1bXJ3YWVreHVtdHdha3NhcGhwdjki",
          "index": true
        }
      ]
    }
  ]
}

Query the data that got posted:

$ curl -X GET 'http://localhost:26659/namespaced_data/0c204d39600fddd3/height/10573'
{"data":["8fIMqAB+kQo7+LLmHaDya8oH73hxem6lQWX1"],"height":10573}%