search

omgnetwork / elixir-omg

OMG-Network repository of Watcher and Watcher Info
https://omg.network
Apache License 2.0
211 stars 59 forks source link

Watcher crash loop generates bursts of ethereum calls #1749

Open unnawut opened 3 years ago

unnawut commented 3 years ago

Overview

When a watcher is misconfigured or a bug occurs, the watcher could enter a crash loop and restarts all the processes within. This in turn causes a burst of Ethereum calls to start all the processes. And because the crash is not self-recoverable, the burst could happen every few seconds (5 seconds?). This impacts integrators who relies on Ethereum node as a service e.g. Infura, and causes them to exhaust the quota rapidly.

Actual behaviour

When the watcher is misconfigured or a bug occurs, it brings the watcher into an irrecoverable crash loop and produces a constant burst of Ethereum calls every few seconds.

Expected behaviour

When the watcher is misconfigured or a bug occurs, the watcher should have an incremental backoff period for retries.

InoMurko commented 3 years ago

This is quite a concerning issues. Did this ever occur? Is there a way to replicate this? Have you seen it in the wild and can provide logs?