Wallet Peak Handler thinks time is wrong?

[bradyjarvis]

I am getting this warning message about a dozen times a day on both my farms with the latest build (v0.7.5-11-g814fd32)

[2023-02-22 08:29:33] [ WARNING] --- Wallet peak is in the future, diff: -1 day, 23:59:49.473000 (Are you sure your timezone is set correctly?) (wallet_peak_handler.py:49)

In the debug.log I see three entries for 08:29 2023-02-22T08:29:16.010 wallet chia.wallet.wallet_blockchain: INFO Peak set to: 3285647 timestamp: 1677076183 2023-02-22T08:29:27.931 wallet chia.wallet.wallet_blockchain: INFO Peak set to: 3285648 timestamp: 1677076183 2023-02-22T08:29:32.487 wallet chia.wallet.wallet_blockchain: INFO Peak set to: 3285649 timestamp: 1677076183

chia wallet show looks fine and both wallets are synced. I triple checked my time zones and I have them set manually to CST.

Environment:

• OS: Ubuntu 22.04 and 22.10
• Python version: 3.10.6-1 amd64
• PIP version: 22.2
• Chia version: 1.7.0 and 1.8.0b1
• Chiadog version: v0.7.5-11-g814fd32
• Harvester: local

[martomi]

Thanks for the report! @Artanicus please take a look when you get a chance.

@bradyjarvis meanwhile, to disable the faulty monitoring of the wallet, you can add this (new) section to your config.yaml:

# Remove any service your node isn't running. All are enabled by default.
# Services listed here are checked for health and will start alerting if missing.
monitored_services:
  - FULL_NODE
  - HARVESTER
  - FARMER
  - WALLET

and temporarily remove the WALLET line until this ticket is resolved.

[jinnatar]

There's something really weird there since no TZ mismatch can cause a diff of 2 days, and if it's only intermittent it could be a parsing error.

I'll troubleshoot deeper later today.

[jinnatar]

Here's some extra debugging:

DEBUG:root:Wallet peak 3285647 parsed, logged at 2023-02-22 08:29:16.010000, peak time: 2023-02-22 08:29:43
DEBUG:root:Wallet peak 3285648 parsed, logged at 2023-02-22 08:29:27.931000, peak time: 2023-02-22 08:29:43
DEBUG:root:Wallet peak 3285649 parsed, logged at 2023-02-22 08:29:32.487000, peak time: 2023-02-22 08:29:43
WARNING:root:Wallet peak is in the future, diff: -1 day, 23:59:33.010000 (Are you sure your timezone is set correctly?)
WARNING:root:Wallet peak is in the future, diff: -1 day, 23:59:44.931000 (Are you sure your timezone is set correctly?)
WARNING:root:Wallet peak is in the future, diff: -1 day, 23:59:49.487000 (Are you sure your timezone is set correctly?)

So there's at least a few things here:

1. The size of the future diff is calculated incorrectly. It should be -27 seconds, not almost -2 days. Probably a logic error on my part.
2. You are in fact receiving information from the future :D .. i.e. parsing the peak some 20 seconds before it's been minted. My first guess would be your system time is actually off. Are you running NTP syncing?

I'll see if I can hunt down the error miscalculating the size of the diff. Once that works, we could in theory only raise a WARNING about the clock being in the future by a bit but not consider it a broken wallet.

If you can confirm if the root cause was clock drift that would be great.

[bradyjarvis]

I quadruple checked and my timezone is correct and I am running NTP. I used the new config.yaml and put in my settings. Disabled the wallet handler in one farm, but let it run on the other. Will keep you posted.

[eliliam]

I am also experiencing this issue, I think I noticed it after upgrading the chia client from the one built from source over to the one from the apt repository. Here are the relevant logs from my machine. I also have confirmed that my time is set correctly and node is fully synced as well.

[2023-05-01 11:36:30] [ WARNING] --- Wallet peak is in the future, diff: -1 day, 23:59:44.917000 (Are you sure your timezone is set correctly?) (wallet_peak_handler.py:49)
[2023-05-01 13:53:15] [ WARNING] --- Wallet peak is in the future, diff: -1 day, 23:58:13.711000 (Are you sure your timezone is set correctly?) (wallet_peak_handler.py:49)
[2023-05-01 13:53:24] [ WARNING] --- Wallet peak is in the future, diff: -1 day, 23:58:22.823000 (Are you sure your timezone is set correctly?) (wallet_peak_handler.py:49)
[2023-05-01 13:54:09] [ WARNING] --- Wallet peak is in the future, diff: -1 day, 23:59:06.499000 (Are you sure your timezone is set correctly?) (wallet_peak_handler.py:49)```

[martomi]

I merged the fix from @Artanicus. Let me know if that fixes it.

[bradyjarvis]

I merged the fix from @Artanicus. Let me know if that fixes it.

I updated both farmers last night and it is working so far. Thank you both!

Edit: I should note I am seeing warnings, but they seem reasonable as far as time difference and I assume normal:

[2023-05-03 01:17:49] [WARNING] Wallet peak is in the future, diff: -3s (wallet_peak_handler.py:49)

[2023-05-03 01:17:52] [WARNING] Wallet peak is in the future, diff: -1s (wallet_peak_handler.py:49)

[2023-05-03 01:45:40] [WARNING] Wallet peak is in the future, diff: -9s (wallet_peak_handler.py:49) [2023-05-03 01:45:46] [WARNING] Wallet peak is in the future, diff: -3s (wallet_peak_handler.py:49)

[eliliam]

Mine appears to have gotten better with the fix as well, however I am still seeing a ton of these warnings continually. The time offset is smaller, but still non-negligible. What is this time difference being calculated off of? I'd like to give as much help as I can. I am using NTP and my timezone is correct, as well as running date in the terminal shows the correct time to the second. Here is an excerpt from my chiadog logs, you can see how frequently these are happening, however there are large gaps in the times, so it seems like it's an issue for a few minutes and then resolves itself for an hour or two before popping up again:

[2023-06-08 14:36:42] [ WARNING] --- Wallet peak is in the future, diff: -79s (wallet_peak_handler.py:49)
[2023-06-08 14:36:55] [ WARNING] --- Wallet peak is in the future, diff: -66s (wallet_peak_handler.py:49)
[2023-06-08 14:36:58] [ WARNING] --- Wallet peak is in the future, diff: -62s (wallet_peak_handler.py:49)
[2023-06-08 14:37:08] [ WARNING] --- Wallet peak is in the future, diff: -53s (wallet_peak_handler.py:49)
[2023-06-08 14:37:09] [ WARNING] --- Wallet peak is in the future, diff: -51s (wallet_peak_handler.py:49)
[2023-06-08 14:37:25] [ WARNING] --- Wallet peak is in the future, diff: -36s (wallet_peak_handler.py:49)
[2023-06-08 15:55:38] [ WARNING] --- Wallet peak is in the future, diff: -14s (wallet_peak_handler.py:49)
[2023-06-08 16:52:43] [ WARNING] --- Wallet peak is in the future, diff: -53s (wallet_peak_handler.py:49)
[2023-06-08 16:52:59] [ WARNING] --- Wallet peak is in the future, diff: -37s (wallet_peak_handler.py:49)
[2023-06-08 16:53:23] [ WARNING] --- Wallet peak is in the future, diff: -15s (wallet_peak_handler.py:49)
[2023-06-08 16:53:37] [ WARNING] --- Wallet peak is in the future, diff: -0s (wallet_peak_handler.py:49)
[2023-06-08 18:20:14] [ WARNING] --- Wallet peak is in the future, diff: -15s (wallet_peak_handler.py:49)
[2023-06-08 23:22:17] [ WARNING] --- Wallet peak is in the future, diff: -50s (wallet_peak_handler.py:49)
[2023-06-08 23:22:17] [ WARNING] --- Wallet peak is in the future, diff: -50s (wallet_peak_handler.py:49)
[2023-06-08 23:22:24] [ WARNING] --- Wallet peak is in the future, diff: -43s (wallet_peak_handler.py:49)
[2023-06-08 23:22:42] [ WARNING] --- Wallet peak is in the future, diff: -25s (wallet_peak_handler.py:49)
[2023-06-09 01:39:48] [ WARNING] --- Wallet peak is in the future, diff: -86s (wallet_peak_handler.py:49)
[2023-06-09 01:39:54] [ WARNING] --- Wallet peak is in the future, diff: -81s (wallet_peak_handler.py:49)
[2023-06-09 01:40:31] [ WARNING] --- Wallet peak is in the future, diff: -44s (wallet_peak_handler.py:49)
[2023-06-09 01:41:00] [ WARNING] --- Wallet peak is in the future, diff: -16s (wallet_peak_handler.py:49)
[2023-06-09 01:41:02] [ WARNING] --- Wallet peak is in the future, diff: -15s (wallet_peak_handler.py:49)
[2023-06-09 12:00:08] [ WARNING] --- Wallet peak is in the future, diff: -55s (wallet_peak_handler.py:49)
[2023-06-09 12:00:37] [ WARNING] --- Wallet peak is in the future, diff: -26s (wallet_peak_handler.py:49)
[2023-06-09 12:00:54] [ WARNING] --- Wallet peak is in the future, diff: -10s (wallet_peak_handler.py:49)
[2023-06-09 17:21:21] [ WARNING] --- Wallet peak is in the future, diff: -183s (wallet_peak_handler.py:49)