search

iotaledger / wasp

Node for IOTA Smart Contracts
Apache License 2.0
296 stars 147 forks source link

Implement circuit breakers for off-chain ingest on congestion #1129

Open fijter opened 2 years ago

fijter commented 2 years ago

In case of a bigger set of requests for a chain over what can be handled the queue for SC requests could grow to proportions where a node might run out of memory to keep track of this backlog. In order to avoid this situation we need some kind of circuit breaker at first that will pause all ingestion of new requests through the API/JSON-RPC endpoint until the backlog is under a certain threshold again.

At a later point in time a more sophisticated way of handling congestion would be even better, this will be discussed in a separate issue.

muXxer commented 1 year ago

@fijter we can use a traefik middleware to do the circuit breaking in the node-docker-setup. We only would need an endpoint on node site that gives us some statistics, so we can decide in the middleware to open the circuit.

muXxer commented 1 year ago

See here: https://doc.traefik.io/traefik/middlewares/http/circuitbreaker/

Also maybe interesting for us: https://doc.traefik.io/traefik/middlewares/http/inflightreq/ https://doc.traefik.io/traefik/middlewares/http/ratelimit/