sosreport / sos

A unified tool for collecting system logs and other debug information
http://sos.rtfd.org
GNU General Public License v2.0
507 stars 543 forks source link

ceph_osd is triggered on mon/mgr nodes , and mon/mgr plugins are triggered on osd nodes #3311

Closed nkshirsagar closed 1 year ago

nkshirsagar commented 1 year ago

Looks like the wildcard expression for the files trigger isn't working as expected,

If i have,

files = ('/var/lib/ceph/osd/*',)

with the contents of the non empty /var/lib/ceph/osd/ folder as

root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sos/report/plugins# ls /var/lib/ceph/osd/
ceph-2/                          ceph.client.crash.keyring        ceph.client.osd-removal.keyring  ceph.client.osd-upgrade.keyring 

This does NOT trigger the osd plugin as I would expect.

Also, it seems due to the presense of the empty folders like /var/lib/ceph/osd on mon/mgr nodes, and conversely the presense of empty mon/mgr folders on osd nodes causes the trigger of ceph_mon and ceph_mgr plugins on pure OSD nodes, and same issue for rgw/mgr/mds.

So I think firstly we should not have the files trigger on the folder, but rather non empty folder. But when I tried something as simple as the above code, it did not succeed in triggering the plugin. So I am wondering how to test for non empty folders using wildcards in files.

It seems the ceph plugins are the only ones using wildcards in the files trigger.

ceph_mon, for eg. will trigger like this,

files = ('/var/lib/ceph/mon/', '/var/lib/ceph/*/mon*',
         '/var/snap/microceph/common/data/mon/*')

But the empty folder /var/lib/ceph/mon/ does exist on OSD nodes too when I tested a jammy+quincy setup deployed by juju.

root@juju-215b2d-quincysos-1:/home/ubuntu# ls /var/lib/ceph/
bootstrap-mds  bootstrap-mgr  bootstrap-osd  bootstrap-rbd  bootstrap-rbd-mirror  bootstrap-rgw  crash  mds  mgr  mon  osd  radosgw  tmp
root@juju-215b2d-quincysos-1:/home/ubuntu# 

The initial commit for the ceph plugins used a overridden check_enabled method since iirc the files trigger did not work with wildcards,

+    def check_enabled(self):
+        return True if glob.glob('/var/lib/ceph/mgr/*/*') else False

but that was reverted in commit 6b3c84da03fb9be9be22f88c163d18e3e4397b21 which left the files having the folder names

commit 6b3c84da03fb9be9be22f88c163d18e3e4397b21
Author: Jake Hunsaker <jhunsake@redhat.com>
Date:   Tue Dec 13 15:46:52 2022 -0500

    [ceph_*] Correct enablement trigger for ceph plugins

    The previous changes that broke up the ceph plugin into smaller
    component specific plugins inadvertently broke the plugin enablement by
    overriding the `check_enabled()` function to check for file presence
    under a directory. This in turn replaced the standard checks, such as
    the presence of certain containers.

    Fix this by removing the method override, and leveraging the `files`
    tuple against the component-specific directory into which the globs were
    trying to check.

    Further, update the container name regexes for enablement as they have
    changed slightly since the initial plugin creation.

    Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>

Maybe we can add back the override, and also a call to super(MyPlugin, self).check_enabled() to also trigger the default checks if the override will be triggering the plugin?

Alternately is there a way to make files list work with wildcards? Or then we need some other way to handle this situation where on an OSD node, for eg, the /var/lib/ceph/osd folder contains,

root@juju-215b2d-quincysos-1:/var/lib/ceph/osd# ls
ceph-2  ceph.client.crash.keyring  ceph.client.osd-removal.keyring  ceph.client.osd-upgrade.keyring

and the parent /var/lib/ceph folder contains,

root@juju-215b2d-quincysos-1:/var/lib/ceph# ls
bootstrap-mds  bootstrap-mgr  bootstrap-osd  bootstrap-rbd  bootstrap-rbd-mirror  bootstrap-rgw  crash  mds  mgr  mon  osd  radosgw  tmp

Note the empty folders,

root@juju-215b2d-quincysos-1:/var/lib/ceph# ls mon root@juju-215b2d-quincysos-1:/var/lib/ceph# ls mgr root@juju-215b2d-quincysos-1:/var/lib/ceph# ls mds root@juju-215b2d-quincysos-1:/var/lib/ceph# ls radosgw/ root@juju-215b2d-quincysos-1:/var/lib/ceph#

nkshirsagar commented 1 year ago

As of now, files does not take wildcards, since we end up from https://github.com/sosreport/sos/blob/main/sos/report/plugins/__init__.py#L3290 at

def path_exists(path, sysroot):
    return _os_wrapper(path, sysroot, 'exists')

and further

def _os_wrapper(path, sysroot, method, module=os.path):
    if sysroot and sysroot != os.sep:
        if not path.startswith(sysroot):
            path = os.path.join(sysroot, path.lstrip('/'))
    _meth = getattr(module, method)
    return _meth(path)

Would something like this in utilities.py

def _os_wrapper(path, sysroot, method, module=os.path):
    if sysroot and sysroot != os.sep:
        if not path.startswith(sysroot):
            path = os.path.join(sysroot, path.lstrip('/'))
    _meth = getattr(module, method)
    if method == 'exists' and path[-1]='*':
        return True if glob.glob(path) else False
    return _meth(path)

make sense, at least to handle wildcards at the end, so we can pass for eg in files stuff like '/var/lib/ceph/mon/*' to check for non empty folder of mon, for eg to determine its a mon node.

nkshirsagar commented 1 year ago

testing

testing on OSD nodes:

root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos# bin/sos report

sosreport (version 4.5.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:lxd] skipped command 'lxd.buginfo': required kmods missing: iptable_filter, ip6_tables, ebtables, ip6table_nat, iptable_raw, ebtable_filter, iptable_nat, ip6table_filter, iptable_mangle, ip6table_raw, ip6table_mangle, bpfilter.  
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, udp_diag, xsk_diag, tcp_diag, af_packet_diag, unix_diag, netlink_diag.   Use '--allow-system-changes' to enable collection.
[plugin:ufw] skipped command 'ufw status numbered': required kmods missing: iptable_filter, bpfilter.  
[plugin:ufw] skipped command 'ufw app list': required kmods missing: iptable_filter, bpfilter.  
 Running plugins. Please wait ...

  Finishing plugins              [Running: ubuntu]                                        
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
    /tmp/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq.tar.xz

 Size   2.97MiB
 Owner  root
 sha256 121d06e6c3fc9d8b98c9f22a165e960a90366d16021206dc0a873dd1ad5ba0bd

Please send this file to your support representative.

root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos# tar -xf /tmp/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq.tar.xz
cd root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos# cd sosreport-juju-215b2d-quincysos-1-2023-07-19-
bash: cd: sosreport-juju-215b2d-quincysos-1-2023-07-19-: No such file or directory
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos# cd sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# ls
boot  df         environment  free      installed-debs   ip_addr   last  lsb-release  lsof   mount  ps      root-symlinks  sos_commands  sos_reports  uname   usr  version.txt
date  dmidecode  etc          hostname  installed-snaps  ip_route  lib   lsmod        lspci  proc   pstree  run            sos_logs      sys          uptime  var  vgdisplay
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# cd sos_commands/
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# ls
alternatives  apt    btrfs        cgroups     cron  devicemapper  filesys          grub2     i18n   kernel     login      lvm2    multipath   pam   process    release   snap     sysvipc  usb
apparmor      block  ceph_common  cloud_init  date  devices       firewall_tables  hardware  iscsi  libraries  logrotate  md      networking  pci   processor  scsi      ssh      ubuntu
apport        boot   ceph_osd     console     dbus  dpkg          fwupd            host      juju   libvirt    logs       memory  nvme        perl  python     services  systemd  udisks
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# ceph
ceph                    ceph-conf               ceph-erasure-code-tool  ceph-mon                ceph-osdomap-tool       ceph-syn                cephfs-journal-tool     
ceph-authtool           ceph-crash              ceph-kvstore-tool       ceph-monstore-tool      ceph-post-file          ceph-volume             cephfs-table-tool       
ceph-bluestore-tool     ceph-create-keys        ceph-mds                ceph-objectstore-tool   ceph-rbdnamer           ceph-volume-systemd     
ceph-clsinfo            ceph-dencoder           ceph-mgr                ceph-osd                ceph-run                cephfs-data-scan        
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# ceph
ceph                    ceph-conf               ceph-erasure-code-tool  ceph-mon                ceph-osdomap-tool       ceph-syn                cephfs-journal-tool     
ceph-authtool           ceph-crash              ceph-kvstore-tool       ceph-monstore-tool      ceph-post-file          ceph-volume             cephfs-table-tool       
ceph-bluestore-tool     ceph-create-keys        ceph-mds                ceph-objectstore-tool   ceph-rbdnamer           ceph-volume-systemd     
ceph-clsinfo            ceph-dencoder           ceph-mgr                ceph-osd                ceph-run                cephfs-data-scan        
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# ceph_^C
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# ls
alternatives  apt    btrfs        cgroups     cron  devicemapper  filesys          grub2     i18n   kernel     login      lvm2    multipath   pam   process    release   snap     sysvipc  usb
apparmor      block  ceph_common  cloud_init  date  devices       firewall_tables  hardware  iscsi  libraries  logrotate  md      networking  pci   processor  scsi      ssh      ubuntu
apport        boot   ceph_osd     console     dbus  dpkg          fwupd            host      juju   libvirt    logs       memory  nvme        perl  python     services  systemd  udisks
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# cd ceph_
ceph_common/ ceph_osd/    
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# cd ceph_
ceph_common/ ceph_osd/    
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# cd ceph_common/
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands/ceph_common# ls
ceph_-v
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands/ceph_common# cd ..
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# cd ceph_osd/
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands/ceph_osd# ls
ceph-volume_lvm_list                                                     ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_historic_slow_ops  ceph_daemon_.var.run.ceph.ceph-osd.2.asok_log_dump
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_bluestore_bluefs_available     ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_mempools           ceph_daemon_.var.run.ceph.ceph-osd.2.asok_objecter_requests
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_config_diff                    ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_op_pq_state        ceph_daemon_.var.run.ceph.ceph-osd.2.asok_ops
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_config_show                    ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_ops_in_flight      ceph_daemon_.var.run.ceph.ceph-osd.2.asok_perf_dump
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_blacklist                 ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_osd_network        ceph_daemon_.var.run.ceph.ceph-osd.2.asok_perf_histogram_dump
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_blocked_ops               ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_reservations       ceph_daemon_.var.run.ceph.ceph-osd.2.asok_status
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_historic_ops_by_duration  ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_watchers           ceph_daemon_.var.run.ceph.ceph-osd.2.asok_version
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands/ceph_osd# cd ..
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands# cd ..
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# ls
boot  df         environment  free      installed-debs   ip_addr   last  lsb-release  lsof   mount  ps      root-symlinks  sos_commands  sos_reports  uname   usr  version.txt
date  dmidecode  etc          hostname  installed-snaps  ip_route  lib   lsmod        lspci  proc   pstree  run            sos_logs      sys          uptime  var  vgdisplay
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# cd ..
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos# cd -
/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# ls
boot  df         environment  free      installed-debs   ip_addr   last  lsb-release  lsof   mount  ps      root-symlinks  sos_commands  sos_reports  uname   usr  version.txt
date  dmidecode  etc          hostname  installed-snaps  ip_route  lib   lsmod        lspci  proc   pstree  run            sos_logs      sys          uptime  var  vgdisplay
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# vi sos_logs/
sos.log  ui.log   
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# vi sos_logs/
sos.log  ui.log   
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# vi sos_logs/sos.log 
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# cat sos_logs/sos.log | grep -vi INFO
2023-07-19 09:25:54,083 WARNING: [plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
2023-07-19 09:25:54,084 WARNING: [plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, udp_diag, xsk_diag, tcp_diag, af_packet_diag, unix_diag, netlink_diag.   Use '--allow-system-changes' to enable collection.
2023-07-19 09:25:54,367 WARNING: [plugin:ufw] skipped command 'ufw status numbered': required kmods missing: iptable_filter, bpfilter.  
2023-07-19 09:25:54,367 WARNING: [plugin:ufw] skipped command 'ufw app list': required kmods missing: iptable_filter, bpfilter.  
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# git log
commit cb7f6dac2194e28cc76e10933a07f71a7ffb76a3 (HEAD -> main, origin/main, origin/HEAD)
Author: root <root@juju-215b2d-quincysos-1.cloud.sts>
Date:   Wed Jul 19 09:22:49 2023 +0000

    [utilities][ceph] wildcard support for paths, fix ceph paths

    This lets files triggers use * and also updates the
    ceph plugins to trigger only if folders are non empty

commit d2a14422ef5771fd704a4c761f26768d4bd9bb7f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Fri Jul 14 12:50:58 2023 +0100

    [plugins] py311 fix for re.M

    Add a common regex for re.MULTILINE so that doesn't need to be defined
    and remove the definitions where `(?m)` is being defined

    Closes: #3301
    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit 5bcdb79cb59edf3169d99227777bf700f04a036e
Author: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>
Date:   Wed Jun 28 07:10:57 2023 +0000

    [plugins/ceph][ubuntu] Add microceph collections

    Collect the microceph data in the ceph plugins.

    Signed-off-by: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>

commit 984abd5cf5b4379508b0fc7d959f17e56577964f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Wed May 31 13:37:17 2023 +0100

    [maas] remove the collection of apache2 logs

    MAAS no longer uses apache2 for it web hosting, so no longer required
    for this plugin

    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit af6de1e1ae4da5f8cc768b2d7b52dc8c55303774
Author: Arif Ali <arif.ali@canonical.com>
Date:   Tue Jun 27 17:49:03 2023 +0100

    [gh_worklow] Build and push snap on release

    Adding workdlow to publish the released sos to the latest/candidate channel
    when the a new release is created via the tag. This ensures less mannual
    intervention for future releases.

root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq# cd sos_commands/ceph_osd/
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands/ceph_osd# ls
ceph-volume_lvm_list                                                     ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_historic_slow_ops  ceph_daemon_.var.run.ceph.ceph-osd.2.asok_log_dump
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_bluestore_bluefs_available     ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_mempools           ceph_daemon_.var.run.ceph.ceph-osd.2.asok_objecter_requests
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_config_diff                    ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_op_pq_state        ceph_daemon_.var.run.ceph.ceph-osd.2.asok_ops
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_config_show                    ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_ops_in_flight      ceph_daemon_.var.run.ceph.ceph-osd.2.asok_perf_dump
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_blacklist                 ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_osd_network        ceph_daemon_.var.run.ceph.ceph-osd.2.asok_perf_histogram_dump
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_blocked_ops               ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_reservations       ceph_daemon_.var.run.ceph.ceph-osd.2.asok_status
ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_historic_ops_by_duration  ceph_daemon_.var.run.ceph.ceph-osd.2.asok_dump_watchers           ceph_daemon_.var.run.ceph.ceph-osd.2.asok_version
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-rlqzigq/sos_commands/ceph_osd# exit
exit

MON node:

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/ceph$ juju ssh ceph-mon/0 sudo -s
root@juju-215b2d-quincysos-0:/home/ubuntu# cd sos/
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# ls
sos
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# cd sos/
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sos# ls
AUTHORS      README.md  man                  requirements.txt  sos       sosreport-juju-215b2d-quincysos-0-2023-07-19-amltmer  sosreport-juju-215b2d-quincysos-0-2023-07-19-hfpuili
LICENSE      bin        plugins_overview.py  setup.py          sos.conf  sosreport-juju-215b2d-quincysos-0-2023-07-19-cruepee  tests
MANIFEST.in  docs       po                   snap              sos.spec  sosreport-juju-215b2d-quincysos-0-2023-07-19-exysgfc  tmpfiles
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sos# cd ..
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# git pull
fatal: not a git repository (or any of the parent directories): .git
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# cd sos
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sos# git pull
Already up to date.
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sos# git log
commit d2a14422ef5771fd704a4c761f26768d4bd9bb7f (HEAD -> main, origin/main, origin/HEAD)
Author: Arif Ali <arif.ali@canonical.com>
Date:   Fri Jul 14 12:50:58 2023 +0100

    [plugins] py311 fix for re.M

    Add a common regex for re.MULTILINE so that doesn't need to be defined
    and remove the definitions where `(?m)` is being defined

    Closes: #3301
    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit 5bcdb79cb59edf3169d99227777bf700f04a036e
Author: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>
Date:   Wed Jun 28 07:10:57 2023 +0000

    [plugins/ceph][ubuntu] Add microceph collections

    Collect the microceph data in the ceph plugins.

    Signed-off-by: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>

commit 984abd5cf5b4379508b0fc7d959f17e56577964f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Wed May 31 13:37:17 2023 +0100

    [maas] remove the collection of apache2 logs

    MAAS no longer uses apache2 for it web hosting, so no longer required
    for this plugin

    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit af6de1e1ae4da5f8cc768b2d7b52dc8c55303774
Author: Arif Ali <arif.ali@canonical.com>
Date:   Tue Jun 27 17:49:03 2023 +0100

    [gh_worklow] Build and push snap on release

    Adding workdlow to publish the released sos to the latest/candidate channel
    when the a new release is created via the tag. This ensures less mannual
    intervention for future releases.

    Update the snap creation for anyone building manually, and need python3-venv
    as a build dependancy.

    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit 59c2660584734af92eca2eae31af3fbf5557f853
Author: Jan Jansky <jjansky@redhat.com>
Date:   Mon Jul 10 13:10:22 2023 +0200

root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sos# git remote -v
origin  https://github.com/sosreport/sos/ (fetch)
origin  https://github.com/sosreport/sos/ (push)
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sos# cd ..
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# ls
sos
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# cd ..
root@juju-215b2d-quincysos-0:/home/ubuntu# ls
sos
root@juju-215b2d-quincysos-0:/home/ubuntu# rm -rf sos
root@juju-215b2d-quincysos-0:/home/ubuntu# git clone https://github.com/nkshirsagar/sos
Cloning into 'sos'...
remote: Enumerating objects: 30184, done.
remote: Counting objects: 100% (9980/9980), done.
remote: Compressing objects: 100% (3328/3328), done.
remote: Total 30184 (delta 6854), reused 6842 (delta 6646), pack-reused 20204
Receiving objects: 100% (30184/30184), 18.42 MiB | 7.82 MiB/s, done.
Resolving deltas: 100% (22075/22075), done.
root@juju-215b2d-quincysos-0:/home/ubuntu# cd sos
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# git log
commit cb7f6dac2194e28cc76e10933a07f71a7ffb76a3 (HEAD -> main, origin/main, origin/HEAD)
Author: root <root@juju-215b2d-quincysos-1.cloud.sts>
Date:   Wed Jul 19 09:22:49 2023 +0000

    [utilities][ceph] wildcard support for paths, fix ceph paths

    This lets files triggers use * and also updates the
    ceph plugins to trigger only if folders are non empty

commit d2a14422ef5771fd704a4c761f26768d4bd9bb7f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Fri Jul 14 12:50:58 2023 +0100

    [plugins] py311 fix for re.M

    Add a common regex for re.MULTILINE so that doesn't need to be defined
    and remove the definitions where `(?m)` is being defined

    Closes: #3301
    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit 5bcdb79cb59edf3169d99227777bf700f04a036e
Author: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>
Date:   Wed Jun 28 07:10:57 2023 +0000

    [plugins/ceph][ubuntu] Add microceph collections

    Collect the microceph data in the ceph plugins.

    Signed-off-by: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>

commit 984abd5cf5b4379508b0fc7d959f17e56577964f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Wed May 31 13:37:17 2023 +0100

    [maas] remove the collection of apache2 logs

    MAAS no longer uses apache2 for it web hosting, so no longer required
    for this plugin

    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit af6de1e1ae4da5f8cc768b2d7b52dc8c55303774
Author: Arif Ali <arif.ali@canonical.com>
Date:   Tue Jun 27 17:49:03 2023 +0100

    [gh_worklow] Build and push snap on release

    Adding workdlow to publish the released sos to the latest/candidate channel
    when the a new release is created via the tag. This ensures less mannual
    intervention for future releases.

root@juju-215b2d-quincysos-0:/home/ubuntu/sos# bin^C
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# ls
AUTHORS  LICENSE  MANIFEST.in  README.md  bin  docs  man  plugins_overview.py  po  requirements.txt  setup.py  snap  sos  sos.conf  sos.spec  tests  tmpfiles
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# vin/sos^C
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# bin/sos report --all-logs

sosreport (version 4.5.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:lxd] skipped command 'lxd.buginfo': required kmods missing: ip6table_raw, ip6table_nat, iptable_filter, ebtable_filter, ip6_tables, bpfilter, iptable_mangle, ebtables, iptable_raw, ip6table_mangle, iptable_nat, ip6table_filter.  
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: af_packet_diag, tcp_diag, inet_diag, udp_diag, netlink_diag, xsk_diag, unix_diag.   Use '--allow-system-changes' to enable collection.
[plugin:ufw] skipped command 'ufw status numbered': required kmods missing: iptable_filter, bpfilter.  
[plugin:ufw] skipped command 'ufw app list': required kmods missing: iptable_filter, bpfilter.  
 Running plugins. Please wait ...

  Finishing plugins              [Running: ceph_mon]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
    /tmp/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze.tar.xz

 Size   43.69MiB
 Owner  root
 sha256 1e4f1ce2233c130f2a88b982d1f6305fa3b9d7733e6b0b8a3afb99d668bc3bef

Please send this file to your support representative.

root@juju-215b2d-quincysos-0:/home/ubuntu/sos# 
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# 
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# tar -xf /tmp/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze.tar.xz
cd sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze    
root@juju-215b2d-quincysos-0:/home/ubuntu/sos# cd sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze# ls
boot  df         environment  free      installed-debs   ip_addr   last  lsb-release  lsof   mount  ps      root-symlinks  sos_commands  sos_reports  uname   usr  version.txt
date  dmidecode  etc          hostname  installed-snaps  ip_route  lib   lsmod        lspci  proc   pstree  run            sos_logs      sys          uptime  var  vgdisplay
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze# cd sos_commands/ceph_
ceph_common/ ceph_mgr/    ceph_mon/    
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze# cd sos_commands/ceph_
ceph_common/ ceph_mgr/    ceph_mon/    
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze# cd sos_commands/ceph_^C
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze# cd sos_logs/
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze/sos_logs# ls
sos.log  ui.log
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze/sos_logs# cat sos.log | grep -vi INFO
2023-07-19 09:29:24,726 WARNING: [plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
2023-07-19 09:29:24,727 WARNING: [plugin:networking] skipped command 'ss -peaonmi': required kmods missing: af_packet_diag, tcp_diag, inet_diag, udp_diag, netlink_diag, xsk_diag, unix_diag.   Use '--allow-system-changes' to enable collection.
2023-07-19 09:29:25,173 WARNING: [plugin:ufw] skipped command 'ufw status numbered': required kmods missing: iptable_filter, bpfilter.  
2023-07-19 09:29:25,174 WARNING: [plugin:ufw] skipped command 'ufw app list': required kmods missing: iptable_filter, bpfilter.  
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-ermcrze/sos_logs# exit
exit
Connection to 10.5.1.41 closed.

RGW:

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/ceph$ juju ssh ceph-radosgw/0 sudo -s
root@juju-215b2d-quincysos-4:/home/ubuntu# ls
root@juju-215b2d-quincysos-4:/home/ubuntu# mkdir nikhil
root@juju-215b2d-quincysos-4:/home/ubuntu# cd nikhil
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil# git clone https://github.com/nkshirsagar/sos
Cloning into 'sos'...
remote: Enumerating objects: 30184, done.
remote: Counting objects: 100% (9980/9980), done.
remote: Compressing objects: 100% (3328/3328), done.
remote: Total 30184 (delta 6854), reused 6842 (delta 6646), pack-reused 20204
Receiving objects: 100% (30184/30184), 18.42 MiB | 2.86 MiB/s, done.
Resolving deltas: 100% (22075/22075), done.
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil# ls
sos
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil# cd sos/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# ls
AUTHORS  LICENSE  MANIFEST.in  README.md  bin  docs  man  plugins_overview.py  po  requirements.txt  setup.py  snap  sos  sos.conf  sos.spec  tests  tmpfiles
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# git log
commit cb7f6dac2194e28cc76e10933a07f71a7ffb76a3 (HEAD -> main, origin/main, origin/HEAD)
Author: root <root@juju-215b2d-quincysos-1.cloud.sts>
Date:   Wed Jul 19 09:22:49 2023 +0000

    [utilities][ceph] wildcard support for paths, fix ceph paths

    This lets files triggers use * and also updates the
    ceph plugins to trigger only if folders are non empty

commit d2a14422ef5771fd704a4c761f26768d4bd9bb7f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Fri Jul 14 12:50:58 2023 +0100

    [plugins] py311 fix for re.M

    Add a common regex for re.MULTILINE so that doesn't need to be defined
    and remove the definitions where `(?m)` is being defined

    Closes: #3301
    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit 5bcdb79cb59edf3169d99227777bf700f04a036e
Author: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>
Date:   Wed Jun 28 07:10:57 2023 +0000

    [plugins/ceph][ubuntu] Add microceph collections

    Collect the microceph data in the ceph plugins.

    Signed-off-by: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>

commit 984abd5cf5b4379508b0fc7d959f17e56577964f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Wed May 31 13:37:17 2023 +0100

    [maas] remove the collection of apache2 logs

    MAAS no longer uses apache2 for it web hosting, so no longer required
    for this plugin

    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit af6de1e1ae4da5f8cc768b2d7b52dc8c55303774
Author: Arif Ali <arif.ali@canonical.com>
Date:   Tue Jun 27 17:49:03 2023 +0100

    [gh_worklow] Build and push snap on release

    Adding workdlow to publish the released sos to the latest/candidate channel
    when the a new release is created via the tag. This ensures less mannual
    intervention for future releases.

root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# ls
AUTHORS  LICENSE  MANIFEST.in  README.md  bin  docs  man  plugins_overview.py  po  requirements.txt  setup.py  snap  sos  sos.conf  sos.spec  tests  tmpfiles
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# bin/sos report 

sosreport (version 4.5.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:lxd] skipped command 'lxd.buginfo': required kmods missing: ip6_tables, iptable_filter, iptable_nat, bpfilter, ip6table_nat, iptable_raw, ebtables, ip6table_filter, ebtable_filter, ip6table_raw, iptable_mangle, ip6table_mangle.  
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: tcp_diag, xsk_diag, udp_diag, netlink_diag, unix_diag, af_packet_diag, inet_diag.   Use '--allow-system-changes' to enable collection.
[plugin:ufw] skipped command 'ufw status numbered': required kmods missing: iptable_filter, bpfilter.  
[plugin:ufw] skipped command 'ufw app list': required kmods missing: iptable_filter, bpfilter.  
 Running plugins. Please wait ...

  Finishing plugins              [Running: ubuntu]                                        
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
    /tmp/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt.tar.xz

 Size   2.84MiB
 Owner  root
 sha256 2d335f7da7598bff1039d0ab327f0241195fd1e0ca4ec6f4a3667351741ef30b

Please send this file to your support representative.

root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# tar -xf /tmp/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt.tar.xz
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# cd sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# cat sos_commands/ceph_common/ceph_-v 
boot/            dmidecode        free             installed-snaps  last             lsmod            mount            pstree           sos_commands/    sys/             usr/             vgdisplay
date             environment      hostname         ip_addr          lib/             lsof             proc/            root-symlinks    sos_logs/        uname            var/             
df               etc/             installed-debs   ip_route         lsb-release      lspci            ps               run/             sos_reports/     uptime           version.txt      
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# cat sos_commands/ceph_common/ceph_-v ^C
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# ps -eaf | grep rgw
ceph       44175       1  0 09:19 ?        00:00:02 /usr/bin/radosgw -f --cluster ceph --name client.rgw.juju-215b2d-quincysos-4 --setuser ceph --setgroup ceph
root       48540   46479  0 09:36 pts/1    00:00:00 grep --color=auto rgw
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# ls /var/lib/ceph/radosgw/ceph-rgw.juju-215b2d-quincysos-4/keyring ^C
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# ls^C
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# ls
AUTHORS  MANIFEST.in  bin   man                  po                setup.py  sos       sos.spec                                              tests
LICENSE  README.md    docs  plugins_overview.py  requirements.txt  snap      sos.conf  sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt  tmpfiles
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# cd sos
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos# ls
__init__.py  __pycache__  archive.py  cleaner  collector  component.py  help  missing.py  options.py  policies  presets  report  utilities.py
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos# cd report/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report# ls
__init__.py  __pycache__  plugins  reporting.py
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report# cd plugins/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report/plugins# ls
__init__.py             cloud_init.py           ebpf.py               ipvs.py             microshift.py         openssl.py               pacemaker.py      rpm.py                   systemtap.py
__pycache__             cman.py                 elastic.py            iscsi.py            microshift_ovn.py     openstack_ansible.py     pam.py            rpmostree.py             sysvipc.py
abrt.py                 cobbler.py              etcd.py               iscsitarget.py      migration_results.py  openstack_aodh.py        pci.py            ruby.py                  targetcli.py
acpid.py                cockpit.py              fapolicyd.py          jars.py             mongodb.py            openstack_barbican.py    pcp.py            s390.py                  teamd.py
activemq.py             collectd.py             fcoe.py               java.py             monit.py              openstack_ceilometer.py  perccli.py        salt.py                  tftpserver.py
alternatives.py         collectl.py             fibrechannel.py       juju.py             mpt.py                openstack_cinder.py      peripety.py       saltmaster.py            tigervnc.py
anaconda.py             composer.py             filesys.py            kata_containers.py  mssql.py              openstack_database.py    perl.py           samba.py                 tomcat.py
anacron.py              conntrack.py            firewall_tables.py    kdump.py            multipath.py          openstack_designate.py   pmem.py           sanlock.py               tuned.py
ansible.py              console.py              firewalld.py          keepalived.py       mvcli.py              openstack_glance.py      podman.py         saphana.py               ubuntu.py
apache.py               container_log.py        flatpak.py            kernel.py           mysql.py              openstack_gnocchi.py     postfix.py        sapnw.py                 udev.py
apparmor.py             containerd.py           foreman.py            kernelrt.py         named.py              openstack_heat.py        postgresql.py     sar.py                   udisks.py
apport.py               containers_common.py    foreman_installer.py  keyutils.py         navicli.py            openstack_horizon.py     powerpath.py      sas3ircu.py              ufw.py
apt.py                  convert2rhel.py         foreman_openscap.py   kimchi.py           networking.py         openstack_instack.py     powerpc.py        scsi.py                  unbound.py
arcconf.py              corosync.py             foreman_proxy.py      kpatch.py           networkmanager.py     openstack_ironic.py      ppp.py            seagate_ses.py           unity.py
ata.py                  crio.py                 freeipmi.py           krb5.py             nfs.py                openstack_keystone.py    procenv.py        selinux.py               unpackaged.py
atomichost.py           cron.py                 frr.py                kubernetes.py       nfsganesha.py         openstack_manila.py      process.py        sendmail.py              usb.py
auditd.py               crypto.py               fwupd.py              kvm.py              nginx.py              openstack_mistral.py     processor.py      services.py              usbguard.py
autofs.py               cs.py                   gcp.py                landscape.py        nis.py                openstack_neutron.py     psacct.py         shmcli.py                validation_framework.py
azure.py                ctdb.py                 gdm.py                ldap.py             nodejs.py             openstack_nova.py        ptp.py            skydive.py               vault.py
bcache.py               cups.py                 gfs2.py               leapp.py            npm.py                openstack_novajoin.py    pulp.py           smartcard.py             vdo.py
block.py                cxl.py                  gluster.py            libraries.py        nscd.py               openstack_octavia.py     pulpcore.py       smclient.py              vdsm.py
boom.py                 date.py                 gluster_block.py      libreswan.py        nss.py                openstack_placement.py   puppet.py         snap.py                  veritas.py
boot.py                 dbus.py                 grafana.py            libvirt.py          ntb.py                openstack_sahara.py      pxe.py            snapper.py               vhostmd.py
btrfs.py                dellrac.py              grub.py               lightdm.py          ntp.py                openstack_swift.py       python.py         snmp.py                  virsh.py
buildah.py              devicemapper.py         grub2.py              lilo.py             numa.py               openstack_tripleo.py     qaucli.py         sos_extras.py            virtwho.py
candlepin.py            devices.py              gssproxy.py           login.py            nvidia.py             openstack_trove.py       qpid.py           soundcard.py             vmware.py
canonical_livepatch.py  dhcp.py                 haproxy.py            logrotate.py        nvme.py               opensvc.py               qpid_dispatch.py  squid.py                 vsftpd.py
ceph_ansible.py         discovery.py            hardware.py           logs.py             nvmetcli.py           openvswitch.py           qt.py             ssh.py                   vulkan.py
ceph_common.py          distupgrade.py          host.py               lstopo.py           oddjob.py             origin.py                quagga.py         ssmtp.py                 watchdog.py
ceph_iscsi.py           dlm.py                  hpasm.py              lustre.py           omnipath_client.py    os_net_config.py         rabbitmq.py       sssd.py                  wireless.py
ceph_mds.py             dmraid.py               hpssm.py              lvm2.py             omnipath_manager.py   ostree.py                radius.py         storageconsole.py        x11.py
ceph_mgr.py             dnf.py                  hts.py                lxd.py              omsa.py               ovirt.py                 rasdaemon.py      storcli.py               xdp.py
ceph_mon.py             docker.py               hyperv.py             maas.py             opencl.py             ovirt_engine_backup.py   rear.py           stratis.py               xen.py
ceph_osd.py             docker_distribution.py  i18n.py               manageiq.py         opencontrail.py       ovirt_hosted_engine.py   redis.py          subscription_manager.py  xfs.py
ceph_rgw.py             dovecot.py              infiniband.py         md.py               opendaylight.py       ovirt_imageio.py         release.py        sudo.py                  xinetd.py
cgroups.py              dpkg.py                 insights.py           megacli.py          opengl.py             ovirt_node.py            rhc.py            sunrpc.py                zfs.py
chrony.py               dracut.py               ipa.py                memcached.py        openhpi.py            ovirt_provider_ovn.py    rhcos.py          symcli.py                zvm.py
cifs.py                 drbd.py                 ipmitool.py           memory.py           openshift.py          ovn_central.py           rhui.py           system.py
clear_containers.py     ds.py                   iprconfig.py          microk8s.py         openshift_ovn.py      ovn_host.py              rhv_analyzer.py   systemd.py
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report/plugins# vi ceph_rgw.py 
Display all 349 possibilities? (y or n)^C
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report/plugins# cd plugi^C
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report/plugins# vi ceph_rgw.py 
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report/plugins# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos/report# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sos# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu# ls
nikhil
root@juju-215b2d-quincysos-4:/home/ubuntu# cd nikhil/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil# ls
sos
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil# cd sos/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# ls
AUTHORS  MANIFEST.in  bin   man                  po                setup.py  sos       sos.spec                                              tests
LICENSE  README.md    docs  plugins_overview.py  requirements.txt  snap      sos.conf  sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt  tmpfiles
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# cd sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# ls
boot  df         environment  free      installed-debs   ip_addr   last  lsb-release  lsof   mount  ps      root-symlinks  sos_commands  sos_reports  uname   usr  version.txt
date  dmidecode  etc          hostname  installed-snaps  ip_route  lib   lsmod        lspci  proc   pstree  run            sos_logs      sys          uptime  var  vgdisplay
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# cd var/lib/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/var/lib# ls
cloud  dbus  juju  systemd
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/var/lib# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/var# vi ^C
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/var# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# cd ..
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# vi sos/report/plugins/ceph_rgw.py 
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos# cd sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt# cd var/log/ceph/
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/var/log/ceph# ls
ceph-client.rgw.juju-215b2d-quincysos-4.log
root@juju-215b2d-quincysos-4:/home/ubuntu/nikhil/sos/sosreport-juju-215b2d-quincysos-4-2023-07-19-anjbvvt/var/log/ceph# exit
exit
Connection to 10.5.2.18 closed.
ubuntu@nkshirsagar-bastion:~/stsstack-bundles/ceph$ 

MICROCEPH:

root@demonax:/home/ubuntu/nikhil# git clone https://github.com/nkshirsagar/sos
Cloning into 'sos'...
remote: Enumerating objects: 30184, done.
remote: Counting objects: 100% (9979/9979), done.
remote: Compressing objects: 100% (3327/3327), done.
remote: Total 30184 (delta 6854), reused 6842 (delta 6646), pack-reused 20205
Receiving objects: 100% (30184/30184), 18.42 MiB | 14.66 MiB/s, done.
Resolving deltas: 100% (22075/22075), done.
root@demonax:/home/ubuntu/nikhil# ls
sos
root@demonax:/home/ubuntu/nikhil# cd sos/
root@demonax:/home/ubuntu/nikhil/sos# ls
AUTHORS  LICENSE  MANIFEST.in  README.md  bin  docs  man  plugins_overview.py  po  requirements.txt  setup.py  snap  sos  sos.conf  sos.spec  tests  tmpfiles
root@demonax:/home/ubuntu/nikhil/sos# bin/sos report^C
root@demonax:/home/ubuntu/nikhil/sos# git log
commit cb7f6dac2194e28cc76e10933a07f71a7ffb76a3 (HEAD -> main, origin/main, origin/HEAD)
Author: root <root@juju-215b2d-quincysos-1.cloud.sts>
Date:   Wed Jul 19 09:22:49 2023 +0000

    [utilities][ceph] wildcard support for paths, fix ceph paths

    This lets files triggers use * and also updates the
    ceph plugins to trigger only if folders are non empty

commit d2a14422ef5771fd704a4c761f26768d4bd9bb7f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Fri Jul 14 12:50:58 2023 +0100

    [plugins] py311 fix for re.M

    Add a common regex for re.MULTILINE so that doesn't need to be defined
    and remove the definitions where `(?m)` is being defined

    Closes: #3301
    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit 5bcdb79cb59edf3169d99227777bf700f04a036e
Author: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>
Date:   Wed Jun 28 07:10:57 2023 +0000

    [plugins/ceph][ubuntu] Add microceph collections

    Collect the microceph data in the ceph plugins.

    Signed-off-by: Nikhil Kshirsagar <nikhil.kshirsagar@canonical.com>

commit 984abd5cf5b4379508b0fc7d959f17e56577964f
Author: Arif Ali <arif.ali@canonical.com>
Date:   Wed May 31 13:37:17 2023 +0100

    [maas] remove the collection of apache2 logs

    MAAS no longer uses apache2 for it web hosting, so no longer required
    for this plugin

    Signed-off-by: Arif Ali <arif.ali@canonical.com>

commit af6de1e1ae4da5f8cc768b2d7b52dc8c55303774
Author: Arif Ali <arif.ali@canonical.com>
Date:   Tue Jun 27 17:49:03 2023 +0100

    [gh_worklow] Build and push snap on release

    Adding workdlow to publish the released sos to the latest/candidate channel
    when the a new release is created via the tag. This ensures less mannual
    intervention for future releases.

root@demonax:/home/ubuntu/nikhil/sos# bin/sos report --all-logs

sosreport (version 4.5.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, netlink_diag, xsk_diag, af_packet_diag, tcp_diag, udp_diag.   Use '--allow-system-changes' to enable collection.
 Running plugins. Please wait ...

  Finishing plugins              [Running: ceph_mon]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
    /tmp/sosreport-demonax-2023-07-19-kwzsgmn.tar.xz

 Size   41.29MiB
 Owner  root
 sha256 6bfccd0e87e2e98750d1125432af7c42e2403018bed373ea14423d57d6170a51

Please send this file to your support representative.

root@demonax:/home/ubuntu/nikhil/sos# tar -xf /tmp/sosreport-demonax-2023-07-19-kwzsgmn.tar.xz
cd root@demonax:/home/ubuntu/nikhil/sos# cd sosreport-demonax-2023-07-19-kwzsgmn
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# ls
boot  dev  dmidecode    etc   hostname        installed-snaps  ip_route  lib          lsmod  lspci  proc  pstree         run           sos_logs     sys    uptime  var          vgdisplay
date  df   environment  free  installed-debs  ip_addr          last      lsb-release  lsof   mount  ps    root-symlinks  sos_commands  sos_reports  uname  usr     version.txt
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd sos_commands/ceph_
ceph_common/ ceph_mon/    
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd sos_commands/ceph_
ceph_common/ ceph_mon/    
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd sos_commands/ceph_^C
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# ps -af | grep ^C
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# ps -eaf | grep osd
root       34627       1  0 Jun02 ?        00:00:00 /bin/sh /snap/microceph/338/commands/osd.start
root       36645       1  0 Jun02 ?        02:48:30 ceph-osd --cluster ceph --id 0
root       38275       1  0 Jun02 ?        02:50:50 ceph-osd --cluster ceph --id 1
root       39929       1  0 Jun02 ?        02:50:19 ceph-osd --cluster ceph --id 2
root      215368  208345  0 09:39 pts/3    00:00:00 grep --color=auto osd
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd ..
root@demonax:/home/ubuntu/nikhil/sos# ls
AUTHORS  LICENSE  MANIFEST.in  README.md  bin  docs  man  plugins_overview.py  po  requirements.txt  setup.py  snap  sos  sos.conf  sos.spec  sosreport-demonax-2023-07-19-kwzsgmn  tests  tmpfiles
root@demonax:/home/ubuntu/nikhil/sos# vi sos/report/plugins/ceph_osd.py 
root@demonax:/home/ubuntu/nikhil/sos# cd -
/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/
lxd/       microceph/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/
lxd/       microceph/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/
lxd/       microceph/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/
lxd/       microceph/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/microceph/
338/     common/  current/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/microceph/
338/     common/  current/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/microceph/
338/     common/  current/ 
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn# cd var/snap/microceph/common/
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common# ls
data  logs
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common# cd data/
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common/data# ls
mon  osd
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common/data# cd osd/
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common/data/osd# ls
ceph-0  ceph-1  ceph-2
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common/data/osd# cd ceph-0/
root@demonax:/home/ubuntu/nikhil/sos/sosreport-demonax-2023-07-19-kwzsgmn/var/snap/microceph/common/data/osd/ceph-0# ls
bfm_blocks  bfm_blocks_per_key  bfm_bytes_per_block  bfm_size  block  bluefs  ceph_fsid  fsid  kv_backend  magic  mkfs_done  ready  require_osd_release  type  whoami