shatteredsilicon / ssm-submodules

GNU Affero General Public License v3.0
1 stars 2 forks source link

Additional Security Audit Scanning #31

Closed gordan-bobic closed 1 year ago

gordan-bobic commented 2 years ago

Add a build feature that performs a verifivation of packages from RPMs to reduce scanning burden and extend scanning. Get full list of packages: rpm -qa For each package, rpm --verify Anything that gets reported means the checksum doesn't match what is in the package. This list should be a separate build log artifact that can be manually checked.

ONLY for the files that: 1) fail the checksum OR 2) get flagged by clamscan or rkhunter 3) aren't owned by any package (get all files using find then for each file check what package owns it using rpm -qf filename, if rpm says nothing owns it, add to the list)

add scanning using virustotal.

Reason for limiting virustotal scanning is because we only have a standard account which limits us to 4 requests per minute / 500 requests per day / 15,500 requests per month. So we only want to scan things that have some indication of possibility of something not being quite right.

You can get VT packages from here (rebuild on aarch64 if necessary): https://ftp.redsleeve.org/pub/misc/vt/

VT api key was forwarded privately. Make sure this doesn't make it into anything pushed into git repositories.

To scan using vt:

vt scan file /path/to/file that submits it, but it doesn't block until it is scanned. It will return a hash. You then check every 10 seconds using:

vt analysis big_long_hash_returned_above

Build time audit log artifact for human operator review should include:

1) Files not owned by any package (simple list) 2) Files flagged by clamscan (full output for each file flagged) 3) Files flagged by rkhunter (output for each file flagged) 4) Files flagged by vt (full output for each file where malicious or suspicious flags returned by vt analysisare greater than 0

oblitorum commented 1 year ago

I'm gonna combine all of them into one file, is this looks ok? Example log artifact logs/security-audit-scanning.log:

rkhunter scanning
=================

Warning: Checking for prerequisites               [ Warning ]
The file of stored file properties (rkhunter.dat) does not exist, and should be created. To do this type in 'rkhunter --propupd'.
Warning: WARNING! It is the users responsibility to ensure that when the '--propupd' option
is used, all the files on their system are known to be genuine, and installed from a
reliable source. The rkhunter '--check' option will compare the current file properties
against previously stored values, and report if any values differ. However, rkhunter
cannot determine what has caused the change, that is for the user to do.
Warning: The command '/usr/bin/egrep' has been replaced by a script: /usr/bin/egrep: POSIX shell script, ASCII text executable
Warning: The command '/usr/bin/fgrep' has been replaced by a script: /usr/bin/fgrep: POSIX shell script, ASCII text executable
Warning: The kernel modules directory '/lib/modules' is missing or empty.
Warning: No running system logging daemon has been found.

clamav scanning
===============

/tmp/xxx.tar: (HEX)XXX.TEST.3.UNOFFICIAL FOUND
/tmp/xxx.cab: (HEX)XXX.TEST.3.UNOFFICIAL FOUND

note: files not owned by any package and flagged by vt will be append to the end of the log artifact file.

oblitorum commented 1 year ago

Updated example log artifact logs/security-audit-scanning.log:

rkhunter scanning
=================

Warning: Checking for prerequisites               [ Warning ]
The file of stored file properties (rkhunter.dat) does not exist, and should be created. To do this type in 'rkhunter --propupd'.
Warning: WARNING! It is the users responsibility to ensure that when the '--propupd' option
is used, all the files on their system are known to be genuine, and installed from a
reliable source. The rkhunter '--check' option will compare the current file properties
against previously stored values, and report if any values differ. However, rkhunter
cannot determine what has caused the change, that is for the user to do.
Warning: The command '/usr/bin/egrep' has been replaced by a script: /usr/bin/egrep: POSIX shell script, ASCII text executable
Warning: The command '/usr/bin/fgrep' has been replaced by a script: /usr/bin/fgrep: POSIX shell script, ASCII text executable
Warning: The kernel modules directory '/lib/modules' is missing or empty.
Warning: No running system logging daemon has been found.

clamav scanning
===============

/tmp/xxx.tar: (HEX)XXX.TEST.3.UNOFFICIAL FOUND
/tmp/xxx.cab: (HEX)XXX.TEST.3.UNOFFICIAL FOUND

package checksum verification
=============================

S.5....T.    /etc/dnf/vars/infra
.M.......    /
missing     /boot
.M.......  c /etc/machine-id
......G..  g /var/run/utmp

orphan file scanning
====================

/proc/fb
/proc/fs
/proc/fs/xfs
/proc/fs/xfs/xqm
/proc/fs/xfs/stat
/proc/fs/xfs/xqmstat
/proc/fs/nfsd
/proc/bus
/proc/bus/pci
/proc/bus/pci/0000:00
/proc/bus/pci/0000:00/00.0

It seems that there are always some system files that are not belong to any package, should we ignore those files? (Not sure how to identify those system files)

gordan-bobic commented 1 year ago

Anything under /proc, /sys and /dev can be filtered from the list because those are all virtual file systems. Directories can also be ignored, e.g. /, /boot.

oblitorum commented 1 year ago

OK, got it

oblitorum commented 1 year ago

There are so many orphan files (files aren't owned by any package) in current ssm-server, about 3000+ (example files listed below), that will definitely exceed the vt api limit (I tried with my account api key), maybe we should ignore more files (e.g. files in directories /var/lib/mysql, /var/lib/grafana, /var/lib/rkhunter and etc)

/var/lib/dnf/modulefailsafe/mariadb:10.3:aarch64.yaml
/var/lib/dnf/repos/epel-7ef3a93dd93fc4ce/countme
/var/lib/dnf/repos/baseos-a4ba83fa826b3b88/countme
/var/lib/dnf/repos/epel-modular-d0f259bd98fc025a/countme
/var/lib/dnf/repos/appstream-7992bedf6cf283a9/countme
......
/var/lib/mysql/mysql/columns_priv.MYD
/var/lib/mysql/mysql/time_zone_leap_second.frm
/var/lib/mysql/mysql/general_log.CSV
/var/lib/mysql/mysql/help_category.MYI
/var/lib/mysql/mysql/time_zone_leap_second.MYD
/var/lib/mysql/mysql/time_zone_name.frm
......
/var/lib/rkhunter/db/suspscan.dat
/var/lib/rkhunter/db/rkhunter.dat
/var/lib/rkhunter/db/i18n/zh.utf8
/var/lib/rkhunter/db/i18n/tr
/var/lib/rkhunter/db/i18n/zh
......
gordan-bobic commented 1 year ago

/var/lib/dnf can be ignored, it's the yum/dnf metadata database /var/lib/mysql can be ignored, that is mysql database files /var/lib/rkhunter can be ignored, that is rkhunter metadata

Most of /var/cache content is probably also ignorable.

oblitorum commented 1 year ago

OK, I got it, there are other reasonable directories such as /var/lib/grafana/, /var/lib/spool, I'll ignore those directories too.

gordan-bobic commented 1 year ago

Grafana yes, for spool, check where crons are, that should probably be checked because some malware could potentially put something in there.

oblitorum commented 1 year ago

OK

oblitorum commented 1 year ago

Current ignored directories:

/proc/*
/dev/*
/sys/*
/var/lib/mysql/*
/var/lib/rkhunter/*
/var/lib/grafana/*
/var/log/*
/root/*
/home/*
/opt/ss/*
/tmp/*
/opt/prometheus/*
/run/*.pid

Let me know if any of them should not be ignored. And there are 79 orphan files left in several directories, I'm not very sure if we need to ignore them or just leave it be.

/var/spool/mail/ssm
/var/spool/mail/grafana
/var/lib/dnf/history.sqlite-shm
/var/lib/dnf/history.sqlite-wal
/var/lib/dnf/history.sqlite
/var/lib/dnf/modulefailsafe/nginx:1.14:aarch64.yaml
/var/lib/dnf/modulefailsafe/perl-DBI:1.641:aarch64.yaml
/var/lib/dnf/modulefailsafe/python36:3.6:aarch64.yaml
/var/lib/dnf/modulefailsafe/perl-DBD-MySQL:4.046:aarch64.yaml
/var/lib/dnf/modulefailsafe/perl-libwww-perl:6.34:aarch64.yaml
/var/lib/dnf/modulefailsafe/python27:2.7:aarch64.yaml
/var/lib/dnf/modulefailsafe/perl-IO-Socket-SSL:2.066:aarch64.yaml
/var/lib/dnf/modulefailsafe/python39:3.9:aarch64.yaml
/var/lib/dnf/modulefailsafe/mariadb:10.3:aarch64.yaml
/var/lib/dnf/repos/epel-7ef3a93dd93fc4ce/countme
/var/lib/dnf/repos/baseos-a4ba83fa826b3b88/countme
/var/lib/dnf/repos/appstream-7992bedf6cf283a9/countme
/var/lib/dnf/repos/epel-modular-d0f259bd98fc025a/countme
/var/lib/dnf/repos/extras-51010d403f6dc12e/countme
/var/lib/alternatives/python
/var/lib/alternatives/libnssckbi.so
/var/lib/alternatives/ld
/var/lib/alternatives/python3
/var/lib/rpm/Filetriggername
/var/lib/rpm/.dbenv.lock
/var/lib/rpm/Recommendname
/var/lib/rpm/Enhancename
/var/lib/rpm/.rpm.lock
/var/lib/rpm/Supplementname
/var/lib/rpm/Transfiletriggername
/var/lib/rpm/Suggestname
/var/lib/cloud/scripts/per-once/init-mysql-password
/var/lib/cloud/scripts/per-boot/generate-ssl-certificate
/.dockerenv
/opt/entrypoint.sh
/opt/playbook-init.yml
/opt/playbook-install.yml
/srv/nginx/certificate.key
/srv/nginx/certificate.crt
/srv/nginx/ca-certs.pem
/srv/nginx/dhparam.pem
/srv/nginx/certificate.conf
/usr/share/grafana/public/img/ssm-logo.png
/usr/share/fonts/urw-base35/.uuid
/usr/share/fonts/.uuid
/usr/share/fonts/dejavu/.uuid
/usr/lib/fontconfig/cache/CACHEDIR.TAG
/usr/lib/fontconfig/cache/bbffa04a-4961-4179-84df-793b3c08a567-le64.cache-7
/usr/lib/fontconfig/cache/abfebadd-206f-4b49-bc9c-f17d951a79bf-le64.cache-7
/usr/lib/fontconfig/cache/6b981ddc-ae10-49cc-aa46-b99ef01fe570-le64.cache-7
/etc/shadow-
/etc/resolv.conf
/etc/group-
/etc/yum.repos.d/local.repo
/etc/yum.repos.d/ssm.repo
/etc/adjtime.rpmnew
/etc/sysconfig/sshd-permitrootlogin
/etc/dnf/modules.d/mariadb.module
/etc/dnf/modules.d/perl-libwww-perl.module
/etc/dnf/modules.d/python27.module
/etc/dnf/modules.d/nginx.module
/etc/dnf/modules.d/perl-DBI.module
/etc/dnf/modules.d/perl-DBD-MySQL.module
/etc/dnf/modules.d/perl-IO-Socket-SSL.module
/etc/dnf/modules.d/python39.module
/etc/dnf/modules.d/python36.module
/etc/passwd-
/etc/.pwd.lock
/etc/nsswitch.conf.bak
/etc/subgid-
/etc/vconsole.conf
/etc/subuid-
/etc/BUILDTIME
/etc/rpm/macros.image-language-conf
/etc/grafana/.bash_logout
/etc/grafana/.bash_profile
/etc/grafana/.bashrc
/etc/gshadow-
/run/cron.reboot
gordan-bobic commented 1 year ago

/opt/ss is kinda important, the main thing we need to make sure is clean, so I think we should include those.

The following can be ignored: /usr/lib/fontconfig/cache /var/lib/dnf /var/lib/rpm /var/spool/mail

Why is grafana's $HOME in /etc/grafana/? That doesn't seem right. Can we make it /var/lib/grafana?

What is in /etc/yum.repos.d/local.repo ? That shouldn't be present on release builds.

oblitorum commented 1 year ago

Yes, we can make grafana's $HOME to /var/lib/grafana, it's currently defined in ansible playbook:

    - name: SSM                        | Create grafana user
      user:
        uid: 472
        name: grafana
        group: grafana
        home: /etc/grafana
        shell: /sbin/nologin

/etc/yum.repos.d/local.repo is for those sub packages it just build (grafana, prometheus, ssm-dashboards etc), so that it can install them with yum/dnf, it's disabled by default.

    - name: SSM                        | Add local YUM repository
      when: ansible_os_family == 'RedHat'
      copy:
        dest: /etc/yum.repos.d/local.repo
        content: |
          [local]
          name=Local repo
          baseurl=file:///tmp/RPMS
          gpgcheck=0
          enabled=0

    - name: SSM                        | Install RPMs
      when: ansible_os_family == 'RedHat'
      shell: microdnf -y --enablerepo local install --nodocs --noplugins --best {{ item }}
      with_items:
        - nginx
        - consul
        - grafana
        - prometheus
        - orchestrator
        - ssm-qan-api
        - ssm-qan-app
        - percona-toolkit-ssm-minimal
        - ssm-dashboards
        - ssm-server
        - ssm-client
        - ssm-manage
        - ssm-managed
        - rds_exporter
        - mariadb
        - mariadb-server