TurboTurtle / rig

A lightweight, flexible, easy to use system monitoring and event handling utility
GNU General Public License v2.0
10 stars 7 forks source link

rig trigger command 'yum update' fails to generate sosreport. #29

Closed amtdas closed 4 years ago

amtdas commented 4 years ago

Scenario: Deploy a logs rig that watches for a specific, exact message, then trigger the rig. Expected: Sosreport should be generated whenever kernel is updated using command 'yum update kernel'.

Steps [1] Created rig trigger for command 'yum update kernel' and watch for a specific messages 'yum update kernel' in /var/log/dnf.log. [2] Completed kernel upgrade with command #yum update kernel [3] /var/log/dnf.log shows message entry of 2020-11-11T12:49:19Z DDEBUG Command: yum update kernel [4] sosreport didnot generated at its default location /var/tmp or /var/tmp/rig/sxavr/.

[root@rh83-new rig]# rig logs --logfile /var/log/dnf.log --message "yum update kernel" --sosreport
sxavr
[root@rh83-new rig]# rig list
ID     PID    Type    Watching                       Trigger                             Status    
====================================================================================================
sxavr  38565  logs    dnf.log, journals: system      yum update kernel                   Running   
ozqmh  37366  logs    dnf.log, journals: system      yum update                          Running   
cbfzw  37348  logs    dnf.log, journals: system      yum update                          Running   
tuvfo  31035  logs    dnf.log, journals: system      yum install                         Running   
[root@rh83-new rig]# 

[root@rh83-new tmp]# rig info -i sxavr
{
    "id": "sxavr",
    "pid": "38565",
    "rig_type": "logs",
    "status": "Running",
    "restart_max": 0,
    "restart_count": 0,
    "cmdline": "/usr/bin/rig logs --logfile /var/log/dnf.log --message yum update kernel --sosreport",
    "debug": false,
    "watch": "dnf.log, journals: system",
    "trigger": "yum update kernel",
    "created": "11/11/20 22:48:23",
    "actions": {
        "sosreport": {
            "name": "sosreport",
            "priority": 100,
            "expected_result": "An sosreport from the host in /var/tmp/rig/sxavr/"
        }
    }
}

[root@rh83-new rig]# yum update kernel
<../lines removed../>
Is this ok [y/N]: y
Downloading Packages:
(1/3): kernel-4.18.0-240.1.1.el8_3.x86_64.rpm                                   1.2 MB/s | 4.3 MB     00:03    
(2/3): kernel-modules-4.18.0-240.1.1.el8_3.x86_64.rpm                           2.6 MB/s |  26 MB     00:09    
(3/3): kernel-core-4.18.0-240.1.1.el8_3.x86_64.rpm                              2.7 MB/s |  30 MB     00:11    
-----------------------------------------------------------------------------------------------------------
<../lines removed../>
Installed:
  kernel-4.18.0-240.1.1.el8_3.x86_64   kernel-core-4.18.0-240.1.1.el8_3.x86_64   kernel-modules-4.18.0-240.1.1.el8_3.x86_64                              

#tail -f /var/log/dnf.log
2020-11-11T12:49:17Z INFO --- logging initialized ---
2020-11-11T12:49:17Z DDEBUG timer: config: 2 ms
2020-11-11T12:49:17Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
2020-11-11T12:49:17Z INFO Updating Subscription Management repositories.
2020-11-11T12:49:19Z DEBUG YUM version: 4.2.23
2020-11-11T12:49:19Z DDEBUG Command: yum update kernel 
2020-11-11T12:49:19Z DDEBUG Installroot: /
2020-11-11T12:49:19Z DDEBUG Releasever: 8
2020-11-11T12:49:19Z DEBUG cachedir: /var/cache/dnf
2020-11-11T12:49:19Z DDEBUG Base command: update

[root@rh83-new rig]# ls -lrt  /var/tmp/rig/sxavr/
total 0
[root@rh83-new rig]# 
TurboTurtle commented 4 years ago

This is because --message looks for the specific, exact, string. You'd want to use .*yum update kernel.* to trigger on any line containing that string. If you try that, does the rig collect as you'd expect?

amtdas commented 4 years ago

Can you confirm that the below rig command is correct? I added new rig trigger, updated kernel again but sosreport not generated at "/var/tmp/rig/jnari/"

[root@rh83-new rig]# rig logs --logfile /var/log/dnf.log --message ".*yum update kernel.*" --sosreport
jnari

[root@rh83-new rig]# rig list
ID     PID    Type    Watching                       Trigger                             Status    
====================================================================================================
jnari  53752  logs    dnf.log, journals: system      .*yum update kernel.*               Running   
sxavr  38565  logs    dnf.log, journals: system      yum update kernel                   Running   
ozqmh  37366  logs    dnf.log, journals: system      yum update                          Running   
cbfzw  37348  logs    dnf.log, journals: system      yum update                          Running   
tuvfo  31035  logs    dnf.log, journals: system      yum install                         Running   

[root@rh83-new rig]# rig info -i jnari
{
    "id": "jnari",
    "pid": "53752",
    "rig_type": "logs",
    "status": "Running",
    "restart_max": 0,
    "restart_count": 0,
    "cmdline": "/usr/bin/rig logs --logfile /var/log/dnf.log --message .*yum update kernel.* --sosreport",
    "debug": false,
    "watch": "dnf.log, journals: system",
    "trigger": ".*yum update kernel.*",
    "created": "11/12/20 00:25:48",
    "actions": {
        "sosreport": {
            "name": "sosreport",
            "priority": 100,
            "expected_result": "An sosreport from the host in /var/tmp/rig/jnari/"
        }
    }
}

[root@rh83-new jnari]# yum remove kernel-4.18.0-240.1.1.el8_3

[root@rh83-new jnari]# yum update kernel
<../Lines Removed../>
Transaction test succeeded                              
  Installing       : kernel-4.18.0-240.1.1.el8_3.x86_64 
  Verifying        : kernel-4.18.0-240.1.1.el8_3.x86_64 
Installed products updated.
Installed:
  kernel-4.18.0-240.1.1.el8_3.x86_64                                                                                                                                                                               
Complete!

#tail -f /var/log/dnf.log
2020-11-11T14:32:35Z INFO --- logging initialized ---
2020-11-11T14:32:35Z DDEBUG timer: config: 2 ms
2020-11-11T14:32:35Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
2020-11-11T14:32:35Z INFO Updating Subscription Management repositories.
2020-11-11T14:32:37Z DEBUG YUM version: 4.2.23
2020-11-11T14:32:37Z DDEBUG Command: yum update kernel 
2020-11-11T14:32:37Z DDEBUG Installroot: /
2020-11-11T14:32:37Z DDEBUG Releasever: 8
2020-11-11T14:32:37Z DEBUG cachedir: /var/cache/dnf
2020-11-11T14:32:37Z DDEBUG Base command: update

[root@rh83-new jnari]# ls -lrt /var/tmp/rig/jnari/
total 0
[root@rh83-new jnari]# 
TurboTurtle commented 4 years ago

I see the issue. An attempt to translate sysadmin-familiar globs to pythonic .* matching is biting us here. The choice is to either not do that translation, or switch from using pythonic regex to something else. I'm leaning towards the former.