Multiple log sources shared by NFS to one Chiadog instance - lack of output

[cr8us]

Hi team,

Log sources: Ubuntu 21.04 Chiadog server: Ubuntu 21.04

1. I exported logs folders from a farmer and a couple of harvesters and mount them on Chiadog server.
2. Created respective config files as local customers pointing mounted debug.log
3. tail -f through mounted exports works without anby issues

Unfortunately Chiadog doesnt read logs and reports that farmer and harvs looks offline.

Environment:

• OS: Ubuntu 21.04 on all machines
• Python version: 3.9
• PIP version: 21.1.3 from chiadog
• Chia version: 1.1.7

[martomi]

In general, if tail -f works, I'd expect that chiadog works as well. Can you share your chiadog config and the logs you get from it running?

[cr8us]

Hi martomi,

Thank you for reply. I've attached chiadog by ssh session so used remote log source and have the same issue. Looks like NFS work fine but I have no idea what doesn't.

I get those errors all the time:

[2021-07-14 05:57:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 19509 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:02:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 19809 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:07:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 20109 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:12:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 20409 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:17:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 20709 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:22:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 21010 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:27:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 21310 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:32:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 21610 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:37:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 21910 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:42:43] [ WARNING] --- Your harvester appears to be offline! No events for the past 22210 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:47:44] [ WARNING] --- Your harvester appears to be offline! No events for the past 22510 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:52:44] [ WARNING] --- Your harvester appears to be offline! No events for the past 22810 seconds. (keep_alive_monitor.py:83)
[2021-07-14 06:57:44] [ WARNING] --- Your harvester appears to be offline! No events for the past 23110 seconds. (keep_alive_monitor.py:83)
[2021-07-14 07:02:44] [ WARNING] --- Your harvester appears to be offline! No events for the past 23410 seconds. (keep_alive_monitor.py:83)
[2021-07-14 07:07:44] [ WARNING] --- Your harvester appears to be offline! No events for the past 23710 seconds. (keep_alive_monitor.py:83)
[2021-07-14 07:12:44] [ WARNING] --- Your harvester appears to be offline! No events for the past 24010 seconds. (keep_alive_monitor.py:83)
[2021-07-14 07:17:45] [ WARNING] --- Your harvester appears to be offline! No events for the past 24311 seconds. (keep_alive_monitor.py:83)
[2021-07-14 07:22:45] [ WARNING] --- Your harvester appears to be offline! No events for the past 24611 seconds. (keep_alive_monitor.py:83)
[2021-07-14 07:27:45] [ WARNING] --- Your harvester appears to be offline! No events for the past 24911 seconds. (keep_alive_monitor.py:83)

When I tail -f the logs I see traffic in them, plots showing eligible, everything looks fine. Why chiadog reports offlines for whole environment?

Example config:

`# Please copy this example config to config.yaml
# and adjust it to your needs.

# This is useful to differentiate multiple chiadog
# instances monitoring multiple harvesters
notification_title_prefix: 'harv1'
log_level: INFO

# Only one consumer can be enabled at a time, you can
# delete the section for the consumer which you aren't using
# For Windows file path needs to be absolute.
chia_logs:
  file_log_consumer:
    enable: true
    file_path: '/mnt/harv1_logs/debug.log'
  network_log_consumer:
    enable: false
    remote_file_path: '~/.chia/mainnet/log/debug.log'
    remote_host: ""
    remote_user: ""
    remote_port: 22

# Enable this and chiadog will ping a remote server every 5 minutes
# That way you can know that the monitoring is running as expected
keep_alive_monitor:
  enable_remote_ping: true
  ping_url: 'https://google.pl'

# Enable this and you'll receive a daily summary notification
# on your farm performance at the specified time of the day.
daily_stats:
  enable: true
  time_of_day: 21
  frequency_hours: 24

# We support a lot of notifiers, please check the README for more
# information. You can delete the sections which you aren't using.
# You can also enable more than one notifier and send different
# notifications to each of them. E.g. enable daily_stats only to E-mail.
# If you enable wallet_events you'll get notifications anytime your
# wallet receives some XCH (e.g. farming reward).
notifier:
  pushover:
    enable: false
    daily_stats: true
    wallet_events: true
    credentials:
      api_token: 'dummy_token'
      user_key: 'dummy_key'
  telegram:
    enable: true
    daily_stats: true
    wallet_events: true
    credentials:
      bot_token: 'blablablabal'
      chat_id: 'blablabla'
 smtp:
    enable: false
    daily_stats: true
    wallet_events: true
    credentials:
      sender: 'chia@example.com'
      sender_name: 'chiadog'
      recipient: 'you@example.com'
      username_smtp: 'username'
      password_smtp: 'password'
      host: 'smtp.example.com'
      port: 587
  script:
    enable: false
    daily_stats: true
    wallet_events: true
    script_path: 'tests/test_script.sh'
  discord:
    enable: true
    daily_stats: true
    wallet_events: true
    credentials:
      webhook_url: 'blablabla'
  slack:
    enable: false
    daily_stats: true
    wallet_events: true
    credentials:
      webhook_url: 'https://hooks.slack.com/services/...'
  mqtt:
    enable: false
    daily_stats: true
    wallet_events: true
    topic: chia/chiadog/alert
    qos: 1
    retain: false
    credentials:
      host: '192.168.0.10'
      port: 8883
      username: ''
      password: ''

`

[martomi]

I suggest putting some debug statements here and checking where logs lost along the way. We use Pygtail to read them. See:

https://github.com/martomi/chiadog/blob/6c4832bc72848efcfd0cedcd216d7fade7da7f2d/src/chia_log/log_consumer.py#L76

[cr8us]

Eh, didnt get it. I have to add this lines to the log_sonsumer.py? I am more infra not code guy ;)

[cr8us]

OK, I am out. I have no idea how to debug this and dont want to waste my time for that. Way too much for me, I am pretty busy. Thanks.