search

informalsystems / hermes

IBC Relayer in Rust
https://hermes.informal.systems
Apache License 2.0
433 stars 319 forks source link

Clock Sync Issues #2536

Open shaspitz opened 1 year ago

shaspitz commented 1 year ago

Summary of Bug

I'm expanding integration tests for ICS right now, and continually getting an error from hermes when relaying packets within a docker container. Example output: 

ERROR link error: failed during a client operation: error raised while updating client on chain provider: failed building header with error: light client verification error for chain id provider: invalid light block: header from the future: header_time=2022-08-05T17:32:51.782526888Z now=2022-08-05T17:32:40.274594846Z

Version

I've replicated this bug for both v1.0.0-rc.1 and v0.15.0, with appropriate migration steps taken for the former.

Steps to Reproduce

Can be replicated by running integration tests for the interchain security repo locally (uses docker). See instructions. The PR/brach in which this issue shows up is linked here.

Acceptance Criteria

The "header from the future" error no longer shows up when running hermes for my use case.

Thanks in advance for the help!

For Admin Use

shaspitz commented 1 year ago

After debugging with @jtremback it seems this issue can be solved by removing the custom tendermint config values here, and instead using the default values for timeout_commit and peer_gossip_sleep_duration. Is there some sort of hermes config value (even just a boolean allowing us to bypass these timing checks in hermes) that could be used in our integration test setting? The ICS integration tests can take a while to run, so having the option to tune tendermint parameters to speed up consensus is useful

adizere commented 1 year ago

After debugging with @jtremback it seems this issue can be solved by removing the custom tendermint config values here, and instead using the default values for timeout_commit and peer_gossip_sleep_duration. Is there some sort of hermes config value (even just a boolean allowing us to bypass these timing checks in hermes) that could be used in our integration test setting? The ICS integration tests can take a while to run, so having the option to tune tendermint parameters to speed up consensus is useful

There is a config value in fact! You could try increasing the value for clock_drift, eg make it 15s. The max_block_time is also relevant.

https://github.com/informalsystems/ibc-rs/blob/6bbd272f1d304ed8bda2cb789c14235f57c052c5/config.toml#L181-L191