search

sosreport / sos

A unified tool for collecting system logs and other debug information
http://sos.rtfd.org
GNU General Public License v2.0
512 stars 541 forks source link

sos clean should obfuscate hostnames like dhcp-192-168-0-66 #3388

Open nkshirsagar opened 1 year ago

nkshirsagar commented 1 year ago

As seen in https://access.redhat.com/solutions/5154441 , dhcp often sets hostnames using the numeric parts of the IP address separated by dashes. The cleaner should obfuscate this kind of hostname because the ip address is getting leaked with dashes instead of dots. This has been frequently flagged in Ubuntu autopkgtests.

The sos cleaner regex for IP address is , looking at https://github.com/sosreport/sos/blob/main/sos/cleaner/parsers/ip_parser.py#L15C1-L22C6 ,

class SoSIPParser(SoSCleanerParser): """Handles parsing for IP addresses"""

name = 'IP Parser'
regex_patterns = [
    # IPv4 with or without CIDR
    r'((?<!(-|\.|\d))([0-9]{1,3}\.){3}([0-9]){1,3}(\/([0-9]{1,2}))?)'
]

I think we should also consider non dash characters, to catch leaks like an ip address for eg like w.x.y.z. leaking like something-w-x-y-z when dhcp seems to set the hostnames this way.

pmoravec commented 1 year ago

Generally ACK, I noticed failed avocado tests on testing farms due to such hostnames. While strictly speaking the dhcp-192-168-0-66 is not an IP address (so why to bother with it), it has evidently encoded one :).

We should be careful with the change not to start obfuscating e.g. package names (e.g. sos-4.5.3-5.el8).

arif-ali commented 1 year ago

We're already skipping multiple files, that have package names in the ip_parser.py, maybe further analysis is required, on where else it needs to be skipped

nkshirsagar commented 1 year ago

To make it clear, the failing test reports this, the fact that the hostname (containing the ip address) is present in the journal logs. The test looks like,

        ip_addr=$(ip route show default | awk '/default/ {print $3}')
        if [ "$(grep -rI $ip_addr /tmp/sosreport_test/*)" ]; then
            add_failure "IP address not obfuscated in all places"
            echo "$(grep -rI $ip_addr /tmp/sosreport_test/*)"

It will echo everything that has the ip address, and in this case, since it passes with dashes (grep -rI will take regex so dashes seem to pass)

573s !!! TEST FAILED: IP address not obfuscated in all places !!!
573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager:Oct 06 14:23:25 host0 systemd-hostnamed[571]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager:Oct 06 14:24:33 host0 systemd-hostnamed[454]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager:Oct 06 14:26:47 host0 systemd-hostnamed[453]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager_--boot_-1:Oct 06 14:24:33 host0 systemd-hostnamed[454]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager_--boot:Oct 06 14:26:47 host0 systemd-hostnamed[453]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/var/log/syslog:Oct 6 14:23:25 adt-focal-arm64-sosreport-20231006-142159-juju-4d1272-prod-prop systemd-hostnamed[571]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/var/log/syslog:Oct 6 14:24:33 adt-focal-arm64-sosreport-20231006-142159-juju-4d1272-prod-prop systemd-hostnamed[454]: Changed host name to 'host-10-43-136-14'
573s /tmp/sosreport_test/var/log/syslog:Oct 6 14:26:47 adt-focal-arm64-sosreport-20231006-142159-juju-4d1272-prod-prop systemd-hostnamed[453]: Changed host name to 'host-10-43-136-14'
When systemd-hotnamed changes the hostname to one that has the ip address (though with dashes not dots) , i.e 'host-10-43-136-14', the grep returns true since it thinks it matches the ip address 10.43.136.14 because we don't escape the dots with backslash.

$ cat testgrep
573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager:Oct 06 14:23:25 host0 systemd-hostnamed[571]: Changed host name to 'host-10-43-136-14'
$ grep -rI "10.43.136.14" .
./testgrep:573s /tmp/sosreport_test/sos_commands/logs/journalctl_--no-pager:Oct 06 14:23:25 host0 systemd-hostnamed[571]: Changed host name to 'host-10-43-136-14'
$ grep -rI "10\.43\.136\.14" .

I'm not sure, but should we be obfuscating hostnames anyway?

class SoSHostnameMap(SoSMap):
    """Mapping store for hostnames and domain names

    Hostnames are obfuscated using an incrementing counter based on the total
    number of hosts matched regardless of domain name.

Or is that only when the admin adds it to the hostname map manually before collecting sos?

pmoravec commented 1 year ago

That is exactly the same scheme of the test failure I noticed as well :)

The trick why hostname parser does not clean it is because the parser gets hostnames "just" from hostname output (https://github.com/sosreport/sos/blob/main/sos/cleaner/preppers/hostname.py#L30-L34) and from /etc/hosts (https://github.com/sosreport/sos/blob/main/sos/cleaner/preppers/hostname.py#L49) at the time of running the sos cleaner. But since the hostname was changed from(*) host-10-43-136-14 to something else later on, this string is not understood as a hostname /o.

() at least that was my use case; yours looks like you have changed hostname to* host-10-43-136-14 and run sos cleaner with that hostname "valid"..? That would be strange. But the links above give you pointers what hostnames were treated as hostnames to clean.

nkshirsagar commented 1 year ago

Yes its indeed strange because according to my use-case those logs should have had the hostname obfuscated. The hostname of the system is host-10-43-136-14 iiuc, and it shows up in the logs that hostnamed puts in the journal when it changed to that hostname. Seems like an all out bug then, and not a question of "adding this use case" to the cleaner scenarios because I thought the IP address was leaking, but actually its the hostname thats leaking. But whats even stranger is, if we take a closer look at the debian autopkgtests and what they're doing, the mask tests are,

# test using mask
test_mask () {
    cmd="--mask"
    run_expecting_success "$cmd" extract
    if [ $? -eq 0 ]; then
        if [ ! $(grep host0 /tmp/sosreport_test/hostname) ]; then
            add_failure "hostname not obfuscated with --mask"
        fi
        # we don't yet support binary obfuscation, so skip binary matches
        if [ "$(grep -rI `hostname` /tmp/sosreport_test/*)" ]; then
            add_failure "hostname not obfuscated in all places"
            echo "$(grep -rI `hostname` /tmp/sosreport_test/*)"
        fi
        # only tests first interface
        mac_addr=$(cat /sys/class/net/$(ip route show default | awk '/default/ {print $5}')/address)
        if [ "$(grep -rI $mac_addr /tmp/sosreport_test/*)" ]; then
            add_failure "MAC address not obfuscated in all places"
            echo "$(grep -rI $mac_addr /tmp/sosreport_test/*)"
        fi
        # only tests first interface
        ip_addr=$(ip route show default | awk '/default/ {print $3}')
        if [ "$(grep -rI $ip_addr /tmp/sosreport_test/*)" ]; then
            add_failure "IP address not obfuscated in all places"
            echo "$(grep -rI $ip_addr /tmp/sosreport/_test/*)"
        fi
        update_failures
    fi

So now I am confused about why the first hostname obfuscation scenario checks just for host0 and not the actual hostname, perhaps this ubuntu autopkgtest is out of date and needs updating!

So I tried reproducing this,

root@jammy1:~# ip route show default | awk '/default/ {print $3}'
10.159.14.1

root@jammy1:~# hostname abc-10.159.14.1

root@jammy1:~# hostname
abc-10.159.14.1

root@jammy1:~# cat /var/log/syslog | grep -i hostname
Oct 19 07:52:53 jammy1 systemd-hostnamed[229]: Hostname set to <jammy1> (static)
Oct 19 07:53:23 jammy1 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Oct 19 07:53:51 jammy1 systemd-resolved[181]: System hostname changed to 'abc'.
Oct 19 07:54:02 jammy1 systemd-resolved[181]: System hostname changed to 'abc-10.159.14.1'.

root@jammy1:~# sos report

sosreport (version 4.5.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: xsk_diag, inet_diag, af_packet_diag, unix_diag, netlink_diag, udp_diag, tcp_diag.   Use '--allow-system-changes' to enable collection.
 Running plugins. Please wait ...

  Starting 54/61 system          [Running: lxd processor snap system]                     [plugin:system] _copy_dir: '/proc/sys/fs' Permission denied
  Finishing plugins              [Running: ubuntu]                                        
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
    /tmp/sosreport-abc-10-2023-10-19-jnsxtmn.tar.xz

 Size   2.87MiB
 Owner  root
 sha256 67f7b49ffcdf0f992d298dc8a47bf7d3458f115c37a2ce88a0eeeb810a54d8e6

Please send this file to your support representative.

root@jammy1:~# cd /tmp/

root@jammy1:/tmp# ls -lrt
total 2951
drwx------ 3 root root       3 Oct 19 07:52 systemd-private-d2f1a954d1554f61bd51fe6419c6ffc1-systemd-resolved.service-iaexAj
drwx------ 3 root root       3 Oct 19 07:52 systemd-private-d2f1a954d1554f61bd51fe6419c6ffc1-systemd-logind.service-p9m520
drwx------ 3 root root       3 Oct 19 07:52 snap-private-tmp
drwx------ 3 root root       3 Oct 19 07:54 systemd-private-d2f1a954d1554f61bd51fe6419c6ffc1-systemd-hostnamed.service-Tl2q5f
drwx------ 3 root root       3 Oct 19 07:55 systemd-private-d2f1a954d1554f61bd51fe6419c6ffc1-systemd-timedated.service-nxVbew
-rw------- 1 root root 3008572 Oct 19 07:55 sosreport-abc-10-2023-10-19-jnsxtmn.tar.xz
-rw-r--r-- 1 root root      65 Oct 19 07:55 sosreport-abc-10-2023-10-19-jnsxtmn.tar.xz.sha256
root@jammy1:/tmp# sos clean sosreport-abc-10-2023-10-19-jnsxtmn.tar.xz

sos clean (version 4.5.6)

This command will attempt to obfuscate information that is generally considered
to be potentially sensitive. Such information includes IP addresses, MAC
addresses, domain names, and any user-provided keywords.

Note that this utility provides a best-effort approach to data obfuscation, but
it does not guarantee that such obfuscation provides complete coverage of all
such data in the archive, or that any obfuscation is provided to data that does
not fit the description above.

Users should review any resulting data and/or archives generated or processed by
this utility for remaining sensitive content before being passed to a third
party.

Press ENTER to continue, or CTRL-C to quit.

Found 1 total reports to obfuscate, processing up to 4 concurrently

sosreport-abc-10-2023-10-19-jnsxtmn :              Extracting...
sosreport-abc-10-2023-10-19-jnsxtmn :              Beginning obfuscation...
sosreport-abc-10-2023-10-19-jnsxtmn :              Re-compressing...
sosreport-abc-10-2023-10-19-jnsxtmn :              Obfuscation completed [removed 54 unprocessable files]

Successfully obfuscated 1 report(s)

A mapping of obfuscated elements is available at
    /tmp/sosreport-abc-10-2023-10-19-jnsxtmn-private_map

The obfuscated archive is available at
    /tmp/sosreport-abc-10-2023-10-19-jnsxtmn-obfuscated.tar.xz

    Size    2.75MiB
    Owner   root

Please send the obfuscated archive to your support representative and keep the mapping file private
root@jammy1:/tmp# 
root@jammy1:/tmp# tar -xf /tmp/sosreport-abc-10-2023-10-19-jnsxtmn-obfuscated.tar.xz

root@jammy1:/tmp# cd sosreport-abc-10-2023-10-19-jnsxtmn
root@jammy1:/tmp/sosreport-abc-10-2023-10-19-jnsxtmn# cat hostname 
abc-10.159.14.1

root@jammy1:/tmp/sosreport-abc-10-2023-10-19-jnsxtmn# grep -ri "10.159.14.1" *
hostname:abc-10.159.14.1
proc/sys/kernel/hostname:abc-10.159.14.1
sos_commands/kernel/sysctl_-a:kernel.hostname = abc-10.159.14.1
sos_commands/kernel/uname_-a:Linux abc-10.159.14.1 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
sos_commands/logs/journalctl_--no-pager:Oct 19 07:54:02 abc-10.159.14.1 systemd-resolved[181]: System hostname changed to 'abc-10.159.14.1'.
sos_commands/logs/journalctl_--no-pager:Oct 19 07:54:53 abc-10.159.14.1 crontab[1014]: (root) LIST (root)
sos_commands/logs/journalctl_--no-pager:Oct 19 07:54:54 abc-10.159.14.1 dbus-daemon[198]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.16' (uid=0 pid=1389 comm="hostnamectl status " label="unconfined")
sos_commands/logs/journalctl_--no-pager:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Starting Hostname Service...
sos_commands/logs/journalctl_--no-pager:Oct 19 07:54:54 abc-10.159.14.1 dbus-daemon[198]: [system] Successfully activated service 'org.freedesktop.hostname1'
sos_commands/logs/journalctl_--no-pager:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Started Hostname Service.
sos_commands/logs/journalctl_--no-pager_--boot:Oct 19 07:54:02 abc-10.159.14.1 systemd-resolved[181]: System hostname changed to 'abc-10.159.14.1'.
sos_commands/logs/journalctl_--no-pager_--boot:Oct 19 07:54:53 abc-10.159.14.1 crontab[1014]: (root) LIST (root)
sos_commands/logs/journalctl_--no-pager_--boot:Oct 19 07:54:54 abc-10.159.14.1 dbus-daemon[198]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.16' (uid=0 pid=1389 comm="hostnamectl status " label="unconfined")
sos_commands/logs/journalctl_--no-pager_--boot:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Starting Hostname Service...
sos_commands/logs/journalctl_--no-pager_--boot:Oct 19 07:54:54 abc-10.159.14.1 dbus-daemon[198]: [system] Successfully activated service 'org.freedesktop.hostname1'
sos_commands/logs/journalctl_--no-pager_--boot:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Started Hostname Service.
sos_commands/hardware/lshw:abc-10.159.14.1
sos_commands/host/hostnamectl_status:Transient hostname: abc-10.159.14.1
sos_commands/host/hostname:abc-10.159.14.1
sos_commands/host/hostname_-f:abc-10.159.14.1
sos_commands/docker/docker_info: Name: abc-10.159.14.1
sos_commands/systemd/systemctl_list-machines:  abc-10.159.14.1 (host) running 0      0
sos_commands/systemd/systemd-analyze_plot.svg:graphical.target reached after 6.703s in userspace</text><text x="20" y="30">Ubuntu 22.04.3 LTS abc-10.159.14.1 (Linux 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2) x86-64 lxc</text><g transform="translate(20.000,100)">
sos_commands/systemd/systemctl_status_--all:● abc-10.159.14.1
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Starting Hostname Service...
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Started Hostname Service.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:55 abc-10.159.14.1 systemd[1]: Started snap.lxd.buginfo-5053:4f:53:ec:58:c90d8d2e406d.scope.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: Started Service for snap application lxd.daemon.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: Started /bin/true.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: var-snap-lxd-common-shmounts.mount: Deactivated successfully.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:55:01 abc-10.159.14.1 systemd[1]: snap.lxd.buginfo-5053:4f:53:ec:58:c90d8d2e406d.scope: Deactivated successfully.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:54 abc-10.159.14.1 dbus-daemon[198]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.16' (uid=0 pid=1389 comm="hostnamectl status " label="unconfined")
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:54 abc-10.159.14.1 dbus-daemon[198]: [system] Successfully activated service 'org.freedesktop.hostname1'
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up LVM configuration
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up OVN configuration
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Rotating logs
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up ZFS (2.1)
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Escaping the systemd cgroups
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ====> Detected cgroup V2
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Disabling shiftfs on this kernel (auto)
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: => Re-using existing LXCFS
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2803]: cat: /var/snap/lxd/common/lxcfs.pid: No such file or directory
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: => Starting LXD
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: Started /bin/true.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Starting Hostname Service...
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:54 abc-10.159.14.1 systemd[1]: Started Hostname Service.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:02 abc-10.159.14.1 systemd-resolved[181]: System hostname changed to 'abc-10.159.14.1'.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: Started Service for snap application lxd.daemon.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: Started /bin/true.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: var-snap-lxd-common-shmounts.mount: Deactivated successfully.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2810]: time="2023-10-19T07:54:58Z" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2810]: time="2023-10-19T07:54:58Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2810]: time="2023-10-19T07:54:58Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
sos_commands/systemd/systemctl_status_--all:Oct 19 07:55:00 abc-10.159.14.1 lxd.daemon[2810]: time="2023-10-19T07:55:00Z" level=warning msg="Failed to initialize fanotify, falling back on inotify" err="Failed to watch directory \"/dev\": operation not permitted"
sos_commands/systemd/systemctl_status_--all:Oct 19 07:55:01 abc-10.159.14.1 ovs-vsctl[3022]: ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
sos_commands/systemd/systemctl_status_--all:Oct 19 07:55:01 abc-10.159.14.1 lxd.daemon[2502]: => LXD is ready
sos_commands/systemd/systemctl_status_--all:Oct 19 07:55:01 abc-10.159.14.1 systemd[1]: snap.lxd.buginfo-5053:4f:53:ec:58:c90d8d2e406d.scope: Deactivated successfully.
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up ceph configuration
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up LVM configuration
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up OVN configuration
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Rotating logs
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up ZFS (2.1)
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Escaping the systemd cgroups
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ====> Detected cgroup V2
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Disabling shiftfs on this kernel (auto)
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: => Re-using existing LXCFS
sos_commands/systemd/systemctl_status_--all:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: => Starting LXD
sos_commands/lxd/lxd.buginfo: - Kernel version: Linux abc-10.159.14.1 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
sos_commands/lxd/lxd.buginfo:  server_name: abc-10.159.14.1
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 systemd[1]: Started Service for snap application lxd.daemon.
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: => Preparing the system (24322)
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Loading snap configuration
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up mntns symlink (mnt:[4026533129])
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/storage-pools
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/devices
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up persistent shmounts path
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ====> Making LXD shmounts use the persistent path
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up kmod wrapper
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Preparing /boot
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Preparing a clean copy of /run
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Preparing /run/bin
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:57 abc-10.159.14.1 lxd.daemon[2502]: ==> Preparing a clean copy of /etc
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Preparing a clean copy of /usr/share/misc
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up ceph configuration
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up LVM configuration
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up OVN configuration
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Rotating logs
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Setting up ZFS (2.1)
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Escaping the systemd cgroups
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ====> Detected cgroup V2
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: ==> Disabling shiftfs on this kernel (auto)
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: => Re-using existing LXCFS
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2803]: cat: /var/snap/lxd/common/lxcfs.pid: No such file or directory
sos_commands/lxd/lxd.buginfo:Oct 19 07:54:58 abc-10.159.14.1 lxd.daemon[2502]: => Starting LXD
uname:Linux abc-10.159.14.1 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
var/log/syslog:Oct 19 07:54:02 host0 systemd-resolved[181]: System hostname changed to 'abc-10.159.14.1'.
root@jammy1:/tmp/sosreport-abc-10-2023-10-19-jnsxtmn#

to make sure, I copied the obfuscated sos in a new folder, tar xf'd it,

root@jammy1:/tmp# mv sosreport-abc-10-2023-10-19-jnsxtmn-obfuscated.tar.xz obf/
root@jammy1:/tmp# cd obf/
root@jammy1:/tmp/obf# tar -xf sosreport-abc-10-2023-10-19-jnsxtmn-obfuscated.tar.xz 

root@jammy1:/tmp/obf# ls
sosreport-abc-10-2023-10-19-jnsxtmn  sosreport-abc-10-2023-10-19-jnsxtmn-obfuscated.tar.xz

root@jammy1:/tmp/obf# cd sosreport-abc-10-2023-10-19-jnsxtmn
root@jammy1:/tmp/obf/sosreport-abc-10-2023-10-19-jnsxtmn# cat hostname 
abc-10.159.14.1

What am I missing here? Or is cleaner broken? Is the cleaner NOT supposed to obfuscate the hostname from the logs?

If I look at the private map file,

{
    "hostname_map": {
        "jammy1": "host0"
    },

    },
    "ip_map": {
        "10.159.14.1": "66.86.20.55", <-- [1]
        "10.159.14.0/24": "100.0.0.1/24",
        "10.159.14.0/29": "101.0.0.1/29",
        "10.159.14.0/31": "102.0.0.1/31",

[1] obviously its able to mask this IP as a string but not the IP leaked in the hostname, but more concerning, why isnt the hostname itself obfuscated?
root@jammy1:/tmp/obf/sosreport-abc-10-2023-10-19-jnsxtmn# cat hostname 
abc-10.159.14.1
nkshirsagar commented 1 year ago

@pmoravec Does this look like a cleaner bug to you where in fact the hostname (and not the ip addr) isnt getting obfuscated?

nkshirsagar commented 1 year ago

@TurboTurtle , regarding your comment on IRC, yes it reproduces for a hostname that is "abc-something-not-ipaddress" The cleaner ran on the sos and the cleaned sos contains the hostname I had set as I had set it.

~/sos_test/obfuscated/sosreport-abc-something-not-ipaddress-2023-11-08-olozmdv# cat hostname 
abc-something-not-ipaddress
TurboTurtle commented 1 year ago

Ok, so we have a generalized bug with the hostname parser regex. I'll spend some time over the next few days/weekend to see if I can refine that to address this.

TurboTurtle commented 1 year ago

Oh, actually I see it now.

The hostname in this example (abc-something-not-ipaddress) is not a valid FQDN - there's no domain. In the instance of abc-10.159.14.1 this is also an invalid FQDN as per protocol the TLD cannot be purely numerical, so the regex doesn't capture it.

If you set the hostname to abc-something-not-ipaddress.example(.com), then the parser works as expected.

So this isn't a straight regex issue but rather one in HostnamePrepper with injecting the hostname into the mapping when the contents are a shortname and not an FQDN. There is a theoretical gap here if the TLD has numbers in it, however.