osmosis-labs / osmosis

The AMM Laboratory
https://app.osmosis.zone
Apache License 2.0
887 stars 581 forks source link

Note to Validators #520

Closed faddat closed 2 years ago

faddat commented 2 years ago

Hi,

I'm operating a relayer. If you are operating a validator, it is very important that you:

systemctl enable systemd-timesyncd
systemctl start systemd-timesyncd

please.

If you do not, there are time drift errors, and well, the Hermes default of five seconds is really pretty generous. We should be less than five seconds apart.

faddat commented 2 years ago
Oct 12 09:04:56 archive hermes[741991]:    0: GRPC call return error status status: InvalidArgument, message: "failed to execute message; message index: 0: cannot update client with ID 07-tendermint-1457: failed to verify header: invalid header: new header has a time from the future 2021-10-12 07:04:50.046283195 +0000 UTC (now: 2021-10-12 07:04:43.209109453 +0000 UTC; max clock drift: 5s): invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
ValarDragon commented 2 years ago

Can we raise the max clock drift allowed? Seems unsafe to make the max clock drift under a block time, and you can get block times of 10s with 1-2 round increments

faddat commented 2 years ago

I think that we can raise it to 7s...... maybe?

This is a judgement call which may require prayer / meditation.

Seems like one could, but also seems like this could make IBC overall more risky.

When looking into shared security stuff, I've been imagining a faster parent chain, slower child chain.

These errors, overall, could be taken as declaring "lower block time gonna do better". Maybe. Except when ibcing with a slower chain to whom you may seem to be in the future. Time is harsh, code-wise.

Only way to test this might be to build some interconnected network of chains-- and then grow it.... 🤔

ValarDragon commented 2 years ago

Per that ibc-rs issue, my reading is that increasing this time moderately has ~no risks / downsides. Its mostly about tolerating clock drift on smaller scales. (Or at least thats my reading, I am admittedly horribly under-informed here)

faddat commented 2 years ago

I am admittedly horribly under-informed here

I feel the same. I suppose that we could try it for the next release.

ValarDragon commented 2 years ago

Whats the sitaution like now? I think we're going to be more able to get this resolved via pinned announcements on the discord/telegram. We can coordinate if you'd like to make a discord announcement

faddat commented 2 years ago

This is still happening, but less often, and may have been related to fee calculation in hermes..... but the same thing would happen with the Go relayer, too.

I'm going to close this, and shoot out a missive in discord + survey, to see if validators are using ntp or systemd-timesyncd or similar.

Maybe this was improved in ibc-go 2.0?