Investigate unpredicatable crash

[tiero]

We experience some random crashes in the daemon, no error are logged, both in production and development mode (eg. yarn start)

Most likely due to some bug in the crawler implementation.

[tiero]

The setInterval could lead to some problems

https://stackoverflow.com/questions/6685396/execute-the-setinterval-function-without-delay-the-first-time/6685414#6685414

[tiero]

Why I consider setInterval to be harmful

https://www.zcfy.cc/original/why-i-consider-setinterval-to-be-harmful-zetafleet

[tiero]

https://nodejs.dev/discover-javascript-timers#recursive-settimeout

setInterval starts a function every n milliseconds, without any consideration about when a function finished its execution.

If a function takes always the same amount of time, it's all fine:

setInterval working fine

Maybe the function takes different execution times, depending on network conditions 
... And maybe one long execution overlaps the next one ...
To avoid this, you can schedule a recursive setTimeout to be called when the callback function finishes

[tiero]

@altafan

I think it's clear that 200ms of delay between network calls it's causing a lot of problems. Before moving to a setTimeout solution, I would try to increment the delay like 3000-5000ms first and also trying to catch and skip the execution if any network error happens

[tiero]

Adding a try/catch to the processDeposit and processBalance I was able to avoid a daemon crash and the caught error was a network error, it looked like either the proxy at nigiri.network or esplora APIs banning our client.

Handling the rejection and increasing the interval to 5000ms it seems to have solved the problem for now. I'll keep the daemon running for couple of hours and see what happens