🐞 [Bug]: Nodes received a "Node didn't wake up within 24 hours" error without receiving the wakeup request.

mahendravarmayadala93 commented 3 weeks ago

What happened?

Some nodes received " Node didn't wake up within 24 hours" error without receiving the wakeup request.

4969785170660011757

Could this be because of Time change issue?

which network/s did you face the problem on?

Main

Twin ID/s

No response

Version

No response

Node ID/s

7187, 7189, 7191

Farm ID/s

1745

Contract ID/s

No response

Relevant log output

NA

sabrinasadik commented 3 weeks ago

We need more data pls check with the farmer if they received more logs from the farmerbot monitoring bot for node 7187 - 7189 - 7191

mahendravarmayadala93 commented 3 weeks ago

We need more data pls check with the farmer if they received more logs from the farmerbot monitoring bot for node 7187 - 7189 - 7191

Hi @sabrinasadik ! The client is currently away and unable to provide the log files, as he typically visits his farms on Saturdays. Please let me know if there are any alternative options to address this.

sabrinasadik commented 3 weeks ago

No other screenshots? Did the farmerbot monitor not say anything after that screenshot anymore?

xmonader commented 3 weeks ago

The farmerbot uses local time to schedule node wake-up times. When Daylight Saving Time (DST) shifts occur, the bot's calculations for the daily wakeup may become inaccurate by one hour (given the machine time changes). This can cause nodes to miss their scheduled boot time within the 24-hour window. While this can cause temporary disruption of the daily (24hours wakeup procedures) , it should self-correct the following day. The farmer should still achieve around 95% uptime.

Solutions:

Changing to UTC time in the farmerbot and make all times relative to UTC
Enforce the machine that runs farmerbot to use UTC
23-Hour Check: Adjust the farmerbot's scheduling logic to check for node wake-up every 23 hours instead of 24. This provides a buffer to account for DST transitions and ensures nodes boot consistently.
Farmer-Forced Restart: Instruct farmers to start all their nodes using the farmerbot and then restart the farmerbot itself. This can help realign the scheduling, but since the issue appears to resolve itself, this is not be really needed

Let's prioritize changing to use UTC in the code /or the 23-hour check in the farmerbot's scheduling logic. This seems like the most effective and straightforward solution to address the DST-related issues.

scottyeager commented 3 weeks ago

So if there's a simple way to cause the bot to handle DST changes automatically, I'd say it's worth implementing. I suggested moving to UTC as a way to remove any complications, but I do like the suggestion here of changing from 24 to 23 hours for "urgent" wake ups.

Just for reference, that would be here:

https://github.com/threefoldtech/tfgrid-sdk-go/blob/7c659bbdc337486a5c52fd4632c8d4344e5c5165/farmerbot/internal/farmerbot.go#L400-L406

The issue I see is that this alone might not be enough to actually adjust the cycle. It seems like the bot will continue to wake up the nodes at the old time after the DST shift. That might be acceptable though, and the farmer can intervene to correct the cycle at their convenience.

threefoldtech / tfgrid-sdk-go