search

PowerLoom / snapshotter-lite-v2

Snapshotter Lite: New Protocol
Apache License 2.0
3 stars 1 forks source link

Add support for telegram notifications in V2 lite #29

Closed Seth-Schmidt closed 1 week ago

Seth-Schmidt commented 1 month ago

The current snapshotter lite v2 release does not support telegram notifications for node issues. This feature should be ported from V1 to V2.

Seth-Schmidt commented 1 month ago

Support for TG notifications has been added for lite node v2

xadahiya commented 1 month ago

I tried running this on my local machine but didn't receive any telegram messages for RPC errors (I tried Infura RPC with 0 request quota), sequencer connection failures, and Event processing failures.

Image

I don't know if I'm doing something wrong or the tg alerting service is down. @Seth-Schmidt, let's discuss this when you're up

Seth-Schmidt commented 1 month ago

https://github.com/PowerLoom/snapshotter-lite-v2/issues/29#issuecomment-2275847078

For the Infura RPC requests, setting the rate to 0 will just use the default limits. The free version will throw rate limit warnings for the snapshot processing, but it will ultimately complete without error (assuming no other issues). It will be better to use the API_REQUEST_METHOD allowance to simulate RPC issues. I successfully receive failure notifications when only allowing eth_blockNumber requests so that the initial block number can be set in the dummy event.

Previously, we had not implemented telegram reporting for the dummy snapshot check considering node runners will be present when starting a node. This is why there are no notifications for the failed message send to the sequencer at the end of the simulation submission. However, with the addition of multi-node setups and potential restarts, I can see some value in alerting failure on setup. I've added support for this here: e4cc74f

Event detection failures are harder to simulate, but starting with a bad Prost RPC url will cause event detection to fail and should send an UNHEALTHY_EPOCH_PROCESSING issue message.

Seth-Schmidt commented 1 month ago

Moving target completion to Aug 9 to allow for further review.

Testing instructions have been added to the PR.

Seth-Schmidt commented 4 weeks ago

This review was put on hold with the release of pre-mainnet. Pushing the completion date.

Seth-Schmidt commented 3 weeks ago

Reporting feature has been added and merged to the preliminary dockerify branch for internal testing.

anomit commented 2 weeks ago

Few outstanding issues to be further created for tracking: changing ownership of the bot and making it part of the snapshotter dashboard.