Grafana Dashboard for relayer agents

[serejke]

Hey Hyperlane, we'd really appreciate if you publish a Grafana dashboard JSON config for relayer agents. I know you have an advanced Grafana setup for relayers' monitoring across many envs and chains, and I thought it would be easier to ask you to publish one than developing from scratch.

Validator agents already have one.

Probably this should better be done after https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/1899 to adjust for multi-source relayers.

Could also be integrated with Kurtosis - PR is up that we should integrate to this https://github.com/kurtosis-tech/hyperlane-package/pull/35

Also after completing this, please update the relevant doc page - see this issue https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3194

[nambrot]

Noted, and adding to the backlog!

[nambrot]

Also useful for Rollups as a Service providers

[nambrot]

One metric folks probably want to use is hyperlane_wallet_balance to track the relayer balance on the different keys

[serejke]

We created a simple one like this (relayers-grafana-dashboard.json.zip)

[HariSeldon23]

We have a bot that we use at Pangolin that we've exposed as a Docker image that then checks a wallet's balance and sends an alert to Discord when it's running low on funds.

It's super helpful for us and could easily be extended

Nothing too fancy, but you're welcome to the code if you want it.

[serejke]

I wish you had told us 3 weeks ago 😄 We developed the same monitoring bot, and even made it top-up relayers' wallets from a "bank" account periodically.

Hyperlane team has this funding script, but it requires a refactoring to be used in non-GCP env: https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/typescript/infra/scripts/funding/fund-keys-from-deployer.ts at some point I think it could be moved to a separate repo

[HariSeldon23]

Ah damn! Yeah we can't do top ups as we need Delegation of Authority controls on our treasury and the only way we've managed to achieve this so far is via Fireblocks.

We're also using Azure so we'll need to adjust a few other things to get that up and running

[avious00]

@paulbalaji note that our mates at Kurtosis have a PR up that we should integrate to this https://github.com/kurtosis-tech/hyperlane-package/pull/35 - also after completing this, please update the relevant doc page

[daniel-savu]

Just added a basic dashboard here: https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/tools/grafana/easy-relayer-dashboard-external.json

For now it's quite basic but it has the data we rely most heavily on for alerting, namely the prepare queue. Our most relevant alert is an increasing prepare queue, which means messages are failing to be delivered.

Here's an example screenshot:

hyperlane_wallet_balance is a visualisation still missing, would be great to have a PR with this added