iotaledger / iota-core

Apache License 2.0
59 stars 12 forks source link

shutdown hanging, somehow prometheus related #936

Closed shufps closed 2 months ago

shufps commented 2 months ago

(happened with iota-core alpha.18)

We monitor go metrics like heap and noticed that on some nodes prometheus is not accessible anymore, trying to curl the metrics leads to a timeout.

As expected, in this cases the nodes cannot be shutdown gracefully.

Node was healthy + network-healthy and other things like the info endpoint were working.

Here is a profile of iota-core while it is waiting for Close p2p peer database, P2P, Protocol. pprof.validator-11_20240425-125923_all.zip