search

martomi / chiadog

A watch dog providing a peace in mind that your Chia farm is running smoothly 24/7.
MIT License
457 stars 120 forks source link

WARNING] --- Your harvester appears to be offline! No events for the past 901 seconds. (keep_alive_monitor.py:83) #243

Closed springjools closed 3 years ago

springjools commented 3 years ago

WARNING] --- Your harvester appears to be offline! No events for the past 901 seconds. (keep_alive_monitor.py:83)

I get this error every 5 minutes, even if the harvester is up and running. Sometimes it helps to kill the chiadog process, rm debug.log.offset and restart everything.

Environment:

niebochod commented 3 years ago

I had the same problem after sudden PC reboot, an garbage has been written into logs and log consumer has been confused and throwing exception at start:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 74, in retry_decorator
    logger)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 33, in __retry_internal
    return f()
  File "D:\ProgramData\chiadog\src\chia_log\log_consumer.py", line 77, in _consume_loop
    for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in __init__
    [int(line.strip()) for line in offset_fh]
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in <listcomp>
    [int(line.strip()) for line in offset_fh]
ValueError: invalid literal for int() with base 10: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

It looks like the consumer thread is crashed, so for chiadog it looks like there is no changes in the chia logs, so offline state assumed?

pieterhelsen commented 3 years ago

I had the same problem after sudden PC reboot, an garbage has been written into logs and log consumer has been confused and throwing exception at start:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 74, in retry_decorator
    logger)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 33, in __retry_internal
    return f()
  File "D:\ProgramData\chiadog\src\chia_log\log_consumer.py", line 77, in _consume_loop
    for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in __init__
    [int(line.strip()) for line in offset_fh]
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in <listcomp>
    [int(line.strip()) for line in offset_fh]
ValueError: invalid literal for int() with base 10: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

It looks like the consumer thread is crashed, so for chiadog it looks like there is no changes in the chia logs, so offline state assumed?

This is a known bug for which a fix is in the pipeline. Meanwhile, deleting the debug.log.offset file from the Chiadog directory will fix this issue. More here: https://github.com/martomi/chiadog/discussions/241

Pull request here: https://github.com/martomi/chiadog/pull/214

martomi commented 3 years ago

Bugfix is now released to main in release 0.6.1