Open tkporter opened 5 months ago
I think part of the problem is also that the interface to rocksdb isn't async. So we block when performing rocksdb IO, and we sometimes do this in loops
From https://ryhl.io/blog/async-what-is-blocking/:
To give a sense of scale of how much time is too much, a good rule of thumb is no more than 10 to 100 microseconds between each .await. That said, this depends on the kind of application you are writing. I wonder if it makes sense to move our DB operations to a spawn_blocking closure or something?
There seem to be places where we probably block for wayyy longer than 100 microseconds, like when we call this for the first time upon startup, and it'll loop through tens of thousands of message nonces without ever hitting an .await: https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/dcb67e97da6e0e3d9abd1533dc2a5ca2ff9b4617/rust/agents/relayer/src/msg/processor.rs#L119-L148
as part of this, it might be nice to allow relayer operators to opt out of merkle tree processing
ah that's a good idea. https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3414 is similar - we will no longer block on it, but still will do the work to eventually build the merkle tree. When we get closer to doing this we can consider the stakeholders & whether that's attractiv
as part of this, it might be nice to allow relayer operators to opt out of merkle tree processing
assume this means backfill processing? we still need forward fill merkle tree processing for the multisig ISMs
chatted w/ @daniel-savu - we'll likely do this after the throughput. Plan is to:
Instrumented tokio and was able to confirm that rocks db IO is blocking, and there isn't really anything we can do about avoiding that. The message processor tasks have almost zero idle time even after 5 mins, and merkle processors aren't doing great either:
Rocks db is write optimized and sync, which is essentially the opposite of what we need. Our writes happen after indexing and after confirming a submission, which are network-bound tasks themselves - the gain from having fast writes is almost zero.
On the other hand, we currently do one read for every message ever sent that passes the relayer whitelist (millions at this point). Even after parallelizing the relayer runtime, it takes 8.5 mins to start submitting to high volume chains like Optimism.
We have two DB IO bound processors per chains (message
and merkle_tree
), and 20 chains on the hyperlane context. This means we'd need 40 cores and growing to parallelize each chain, or shard by deploying on different machines. This is more trouble than it's worth for now.
We're opting for a simpler approach now:
@tkporter @daniel-savu when you merge this can you ping @ltyu? syncing on sepolia was taking a long time for him, i think this addresses that
@ltyu this has mostly been fixed, you can use the latest commit on main
(docker image 0cf692e-20240526-164442
)
@tkporter reported that startup seems to be slow again. Only running with a subset of chains seems to fix this, so it's probably due to the high number of chains the omniscient relayer is currently operating. tokio-console
indicates that
the prepare_task
s are the issue , since they take up the most busy time of the runtime, particularly at startup. I wasn't able to narrow this down further, although I suspect that we must be doing some CPU-intensive looping in there.
3 mins into a new relayer run, line 132 (the prepare task - here) takes most of the busy time:
A view into one of the prepare task's lifecycle, showing how it takes up a lot of busy time on startup. With prepare tasks already being >20, it makes sense that some can't be scheduled because the machine doesn't have that many cores.
Problem
Whitelist configuration
log here at 16:56:29 https://cloudlogging.app.goo.gl/XTcjMyFk8jN4DCe38Solution