search

dymensionxyz / roller

CLI tool for deploying Dymension RollApps
72 stars 66 forks source link

RollApp Sequencer migration issues with old sequencer metadata #1097

Open Proskhmer08 opened 3 weeks ago

Proskhmer08 commented 3 weeks ago

Roller version

1.7.2-alpha-rc03

Rollpp type (EVM/WASM) and version

v2.2.0-rc01-56-g147382a

OS

Ubuntu 22.04.5 LTS

Processor Architecture

(GNU/Linux 5.15.0-124-generic x86_64)

Others

System load: 0.0 Memory usage: 1%

Steps to reproduce it

I am migrating my RollApp from Hetzner to a new server.

In the process of importing the previous Sequencer hub key, it created new my_celes_key to fund, which I did. (Although, shouldn't it use the previous key?)

Then it generated new relayer hub key and it asked to fund 20dym for this new relayer instead of using the previous one. I did. (But again, shouldn't it use the previous relayer hub key?)

However, in the process of setting up the relayer and loading IBC channel, it gave me an error msg:

ERROR failed to load active channel, command execution failed: exit status 1, stderr: Error: post failed: Post "https://rpc.hungry-badger-11.telebit.io:443": tls: failed to verify certificate: x509: certificate is valid for telebit.cloud, not rpc.hungry-badger-11.telebit.io

This is the old telebit rpc endpoints from the previous server. I do have the new endpoints, but it never asked for the rpc endpoints during the import. I then went into the rollapp directory and looking to update the sequencer metadata. But there's no init directory inside /.roller/rollapp/.

So then I tried the export feature, roller rollapp sequencer metadata export, but it gave me the error because obviously, there's no init directory. Here's the error msg for that: ERROR failed to export metadata error writing to file: open /home/ubuntu/.roller/rollapp/init/sequencer-metadata.json: no such file or directory

So then I created a new init directory with the sequencer-metadata.json and update it with the new endpoints and percentage fee.

After updating the file, I ran the export command: roller rollapp sequencer metadata export and it asked me to update the metadata with, roller rollapp sequencer metadata update. That seems to work as it gave me this:

SUCCESS transaction succeeded INFO Gas wanted: 200000, Gas used: 147160

However, the rpc endpoints are still the old ones in the response log.

evm_rpcs:
      - https://evm.hungry-badger-11.telebit.io
      explorer_url: ""
      extra_data: null
      gas_price: "2000000000"
      genesis_urls: []
      moniker: ""
      p2p_seeds: []
      rest_api_urls:
      - https://rest.hungry-badger-11.telebit.io
      rpcs:
      - https://rpc.hungry-badger-11.telebit.io
      snapshots: []
  non_critical_extension_options: []
  timeout_height: "0"
signatures: []
do you want to continue? [y/N]: Yes

The metadata.json file already has new endpoints there. I then ran, roller relayer setup, and of course it fails with the same msg as before: ERROR failed to load active channel, command execution failed: exit status 1, stderr: Error: post failed: Post "https://rpc.hungry-badger-11.telebit.io:443": tls: failed to verify certificate: x509: certificate is valid for telebit.cloud, not rpc.hungry-badger-11.telebit.io

So what is the next step to get around this? The sequencer, relayer, da-light-client are offline in the old server for the migration.

Expected result

During the import of the previous sequencer, it should ask for new metadata endpoints. Or if after creating init directory and the metadata.json file, and then running the metadata update command, it should've actually updated with new endpoints.

Actual result

Actual result is unable to migrate sequencer from one server to the next.

loading active IBC channels (0s)
  ERROR   failed to load active channel, command execution failed: exit status 1, stderr: Error: post failed: Post "https://rpc.hungry-badger-11.telebit.io:443": tls: failed to verify certificate: x509: certificate is valid for telebit.cloud, not rpc.hungry-badger-11.telebit.io

Log Files

Light Client Log:

paamr6aaqyaaaaaaaalttpyabaaas
2024-11-01T19:26:39.285Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltd2qab4aar
2024-11-01T19:26:39.293Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltveqaamaar
2024-11-01T19:26:39.307Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltra4absaa5
2024-11-01T19:26:43.218Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaalt6duabwaas
2024-11-01T19:26:43.493Z    ERROR   rpc go-jsonrpc@v0.6.0/server.go:139 RPC Error: Invalid request
2024-11-01T19:26:43.493Z    WARN    rpc go-jsonrpc@v0.6.0/server.go:149 rpc error: Invalid request
2024-11-01T19:26:43.608Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaalt6duaa4aar
2024-11-01T19:26:43.645Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaalt6duabwaac
2024-11-01T19:26:57.233Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltdbiacwabe
2024-11-01T19:26:58.502Z    ERROR   rpc go-jsonrpc@v0.6.0/server.go:139 RPC Error: Invalid request
2024-11-01T19:26:58.502Z    WARN    rpc go-jsonrpc@v0.6.0/server.go:149 rpc error: Invalid request
2024-11-01T19:26:59.323Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltnzqaawaai
2024-11-01T19:27:01.548Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltvkiabsabj
2024-11-01T19:27:04.015Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaalteyqaaiaan
2024-11-01T19:27:04.039Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaalt5daaa6aak
2024-11-01T19:27:06.972Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltv7uaawaaz
2024-11-01T19:27:07.510Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for bagipaamr6aaqyaaaaaaaaltu6aaaeaap
2024-11-01T19:27:07.684Z    ERROR   shwap/bitswap   bitswap/block_fetch.go:190  hasher: no unmarshallers registered for

Relayer log:

--log_level string    The logging level (trace|debug|info|warn|error|fatal|panic) (default "info")
      --trace               print out full stack trace on errors
2024/11/01 18:40:16 something bad happened command execution failed: exit status 1, stderr: Error: post failed: Post "https://rpc.hungry-badger-11.telebit.io:443": tls: failed to verify certificate: x509: certificate is valid for telebit.cloud, not rpc.hungry-badger-11.telebit.io
      --height int         Use a specific height to query state at (this can error if the node is pruning state)
      --log_level string    The logging level (trace|debug|info|warn|error|fatal|panic) (default "info")
      --trace               print out full stack trace on errors
2024/11/01 19:06:09 failed to find connection on the rollapp side for dymages_10094-1: command execution failed: exit status 1, stderr: Error: post failed: Post "https://rpc.hungry-badger-11.telebit.io:443": tls: failed to verify certificate: x509: certificate is valid for telebit.cloud, not rpc.hungry-badger-11.telebit.io
      --height int         Use a specific height to query state at (this can error if the node is pruning state)
      --log_level strin
g    The logging level (trace|debug|info|warn|error|fatal|panic) (default "info")
      --trace               print out full stack trace on errors
2024/11/01 19:06:09 something bad happened command execution failed: exit status 1, stderr: Error: post failed: Post "https://rpc.hungry-badger-11.telebit.io:443": tls: failed to verify certificate: x509: certificate is valid for telebit.cloud, not rpc.hungry-badger-11.telebit.io
      --height int         Use a specific height to query state at (this can error if the node is pruning state)
      --log_level string    The logging level (trace|debug|info|warn|error|fatal|panic) (default "info")
      --trace               print out full stack trace on errors
Screenshot 2024-11-01 at 1 43 54 PM Screenshot 2024-11-01 at 1 43 41 PM

Notes

No response

artemijspavlovs commented 2 weeks ago

hey, thanks for such a thorough issue

shouldn't it use the previous relayer hub key?

good point, will be added, created a separate issue to track: #1104

I tried the export feature, roller rollapp sequencer metadata export, but it gave me the error because there's no init directory. Here's the error msg for that: ERROR failed to export metadata error writing to file: open /home/ubuntu/.roller/rollapp/init/sequencer-metadata.json: no such file or directory

was under the impression that roller creates init during export. Fixed in #1105

During the import of the previous sequencer, it should ask for new metadata endpoints.

it shouldn't. One might want to have the old endpoints available as the sequencer keeps running until the new sequencer is fully prepared. Then perform the update

Or if after creating init directory and the metadata.json file, and then running the metadata update command, it should've actually updated with new endpoints.

unfortunately, I was not able to reproduce a failed metadata update. please try running it again, if the issue persists - leave a comment with the transaction hash here. thx