Support for disabling health check

[adizere]

Crate

relayer-cli

Summary

A feedback from the Cephalopod ops team is that they would like a method to optionally disable the health checkup that Hermes does indiscriminately.

Problem Definition

Whenever Hermes starts up, before running any command, it runs a health checkup mechanism that involves a few rounds of RPC calls to each chain. This functionality is especially useful for users who have not much experience with using Hermes (because the health check can surface problems that would otherwise be hidden cf https://github.com/informalsystems/ibc-rs/issues/697). The functionality, however, is not very useful for power-users such as relayer operators.

The health checkup involves pulling the genesis file, and some chains (e.g., hub-4) have very large genesis files (~100MB). This means that the health check is a liability slowing down the whole ops process.

Proposal

A new option is necessary to allow disabling the health checkup mechanism. This is necessary in particular for CLIs such as:

• tx raw packet-recv,
• packet-ack,
• query unreceived-acks or
• query unreceived-packets.

Acceptance Criteria

• [ ] Disable the health checkup mechanism for all existing CLIs

  • by default, upon starting the runtime for a chain, we should not longer call into the health check-related methods:
  • https://github.com/informalsystems/ibc-rs/blob/85446586daf6dba652271eef31dae29f35da86c2/relayer/src/chain/cosmos.rs#L740-L741

• [ ] Add a new CLI hermes health-check

  • this new CLI should bootstrap chain runtimes for all chains in the config.toml so that the runtimes call the above two methods; thus checking that Hermes has a healthy connection to each chain

    ---

For Admin Use

• [x] Not duplicate issue
• [x] Appropriate labels applied
• [x] Appropriate milestone (priority) applied
• [x] Appropriate contributors tagged
• [x] Contributor assigned/self-assigned

[romac]

It is unclear which method would be best for disabling the health checkup, either a new config.toml option or a global CLI flag. This can be decided as we go along after consulting with Cephalopod.

I think such commands should just never do the health check (but we can add a command that does just that). This could be achieved by supplying an option to the chain runtime when starting it, but there is no need to expose this in the config file or the CLI options (aside perhaps for the start command).

[adizere]

It is unclear which method would be best for disabling the health checkup, either a new config.toml option or a global CLI flag. This can be decided as we go along after consulting with Cephalopod.

I think such commands should just never do the health check (but we can add a command that does just that). This could be achieved by supplying an option to the chain runtime when starting it, but there is no need to expose this in the config file or the CLI options (aside perhaps for the start command).

Perfect timing for this suggestion, Romain. Mircea and I were reading your idea and concluded it would be the best way to go forward. It seems not even the start command would require the health checkup.

Concrete notes after discussing with Cephalopod:

• Between the option of having a global flag for all CLIs (e.g., --no-health-check or a config.toml toggle), best would be to entirely bypass the health check in all CLIs, except for a dedicated new CLI that does solely the health check, e.g., hermes health-check
• the health check proved useful at least on occasions when the operator misconfigured the max_tx_size to exceed the genesis/max_block_bytes parameter, so the check itself is useful

[andynog]

@adizere and @romac, I will look into this one. Also for the health check for the max_block_bytes I'll try to replace the logic. Using the /genesis endpoint might not be optimal because the genesis information might be huge and might time out when retrieving the information, probably a better RPC to get that information is the /consensus_params endpoint like https://rpc-cosmos.cosmostation.io/consensus_params?height=7511042.

This endpoint might provide a better performance and these params can be changed through governance so it's better to take a value close to the latest block height available to ensure it's accurate. The genesis value might not reflect the current value for that parameter.