martomi / chiadog

A watch dog providing a peace in mind that your Chia farm is running smoothly 24/7.
MIT License
460 stars 123 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data #347

Open Nitrooo opened 2 years ago

Nitrooo commented 2 years ago

Chiadog was working fine for days, then it started reporting that my harvester was offline, but that was not the case. This is what I found in the chiadog logs:

...
[2022-07-14 04:26:00] [    INFO] --- Detected 1 skipped signage point.This is expected to happen occasionally and not a reason for concern. (non_skipped_signage_points.py:39)
Exception in thread Thread-1 (_consume_loop):
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/chiadog/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/chiadog/venv/lib/python3.10/site-packages/retry/api.py", line 73, in retry_decorator
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
  File "/chiadog/venv/lib/python3.10/site-packages/retry/api.py", line 33, in __retry_internal
    return f()
  File "/chiadog/src/chia_log/log_consumer.py", line 83, in _consume_loop
    for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):
  File "/chiadog/venv/lib/python3.10/site-packages/pygtail/core.py", line 136, in __next__
    return self.next()
  File "/chiadog/venv/lib/python3.10/site-packages/pygtail/core.py", line 108, in next
    line = self._get_next_line()
  File "/chiadog/venv/lib/python3.10/site-packages/pygtail/core.py", line 280, in _get_next_line
    line = self._filehandle().readline()
  File "/usr/local/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data
[2022-07-14 06:16:37] [ WARNING] --- Your harvester appears to be offline! No events for the past 511 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:21:38] [ WARNING] --- Your harvester appears to be offline! No events for the past 812 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:26:39] [ WARNING] --- Your harvester appears to be offline! No events for the past 1112 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:31:39] [ WARNING] --- Your harvester appears to be offline! No events for the past 1413 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:36:40] [ WARNING] --- Your harvester appears to be offline! No events for the past 1714 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:41:40] [ WARNING] --- Your harvester appears to be offline! No events for the past 2014 seconds. (keep_alive_monitor.py:83)
...

I restarted chiadog's container and it started functioning properly again.

martomi commented 2 years ago

Seems related to https://github.com/martomi/chiadog/issues/327 we should catch that exception, log it, and retry without killing the thread.