peteeckel commented 4 years ago

Describe the bug

I am running Icinga 2 on some CentOS 8.2 machines with SELinux set to enforcing.

When I issue the systemctl reload icinga2 request, I get two AVC messages in the audit log:

[root@medusa38 ~]# ausearch -m AVC -ts boot 
time->Sat Aug 15 13:59:15 2020
type=AVC msg=audit(1597499955.658:120587): avc:  denied  { search } for  pid=473480 comm="icinga2" name="krb5" dev="dm-3" ino=524428 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0
----
time->Sat Aug 15 13:59:15 2020
type=AVC msg=audit(1597499955.669:120588): avc:  denied  { search } for  pid=473480 comm="icinga2" name="krb5" dev="dm-3" ino=524428 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0

The reload completes successfully anyway.

Expected behavior

No AVC messages are logged.

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version):


[root@medusa38 ~]# icinga2 --version 
icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.0-1)

Copyright (c) 2012-2020 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: AlmaLinux release 8.4 (Electric Cheetah) Platform version: 8 (Core) Kernel: Linux Kernel version: 4.18.0-305.10.2.el8_4.x86_64 Architecture: x86_64

Alternate System information: Platform: CentOS Linux Platform version: 8 (Core) Kernel: Linux Kernel version: 4.18.0-193.14.2.el8_2.x86_64 Architecture: x86_64

Build information: Compiler: GNU 8.3.1 Build host: runner-hh8q3bz2-project-322-concurrent-0 OpenSSL version: OpenSSL 1.1.1c FIPS 28 May 2019

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid


* Operating System and version:

[root@medusa38 ~]# cat /etc/redhat-release CentOS Linux release 8.2.2004 (Core)


* Enabled features (`icinga2 feature list`):

[root@medusa38 ~]# icinga2 feature list Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb livestatus opentsdb perfdata statusdata syslog Enabled features: api checker ido-pgsql mainlog notification


* Config validation (`icinga2 daemon -C`):

[root@medusa38 ~]# icinga2 daemon -C [2020-08-15 14:08:16 +0000] information/cli: Icinga application loader (version: 2.12.0-1) [2020-08-15 14:08:16 +0000] information/cli: Loading configuration file(s). [2020-08-15 14:08:16 +0000] information/ConfigItem: Committing config item(s). [2020-08-15 14:08:16 +0000] information/ApiListener: My API identity: medusa38.hindenburgring.com [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 NotificationComponent. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 17 Hosts. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 FileLogger. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 IcingaApplication. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 CheckerComponent. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 17 Zones. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 17 Endpoints. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 ApiUser. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 ApiListener. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 202 CheckCommands. [2020-08-15 14:08:17 +0000] information/ConfigItem: Instantiated 1 IdoPgsqlConnection. [2020-08-15 14:08:17 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2020-08-15 14:08:17 +0000] information/cli: Finished validating the configuration file(s).


## Additional context

The problem can be solved by applying this SELinux policy:

============= icinga2_t ==============

allow icinga2_t krb5_keytab_t:dir search;

Al2Klimov commented 4 years ago

@dgoetz Any idea what's the problem and whether the suggested solution is OK?

peteeckel commented 3 years ago

Any news on this? A customer of mine is planning to migrate to EL8 next year(ish), and while the workaround I sketched is feasible it would be better if it could be integrated in icinga2-selinux (or the underlying problem, if there is one, fixed).

Anything I can do to help?

julianbrost commented 3 years ago

Icinga itself doesn't do anything with Kerberos. I suspect this comes from the PostgreSQL client as that support Kerberos authentication and indeed that so file links against Kerberos libs:

[root@c403f5b4429a /]# ldd /usr/lib64/icinga2/libpgsql_shim.so | grep krb
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fac3a55f000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fac39c04000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fac395d8000)

Al2Klimov commented 3 years ago

Stupid question btw.: isn’t EL7 supported for longer than EL8?

peteeckel commented 3 years ago

Stupid question btw.: isn’t EL7 supported for longer than EL8?

Not quite.

CentOS Linux 8 will be EOL end of this year and will be replaced by CentOS Stream 8. RHEL 8 is unaffected by this change, which is mainly due to a change of release policy in the CentOS project. Which in the opinion of some people renders CentOS dead in the water (naturally IBM/RHEL as the owners of the project strike a different tune).

The place of CentOS Linux 8 as a binary/bug compatible clone of RHEL 8 will be taken by (at least) the community projects AlmaLinux and Rocky Linux, which claim the same support intervals as RHEL.

Given the fact that RHEL7 is substantially behind the leading edge with some packets, migrating to EL8 in whatever flavour makes much sense.

peteeckel commented 3 years ago

Icinga itself doesn't do anything with Kerberos. I suspect this comes from the PostgreSQL client as that support Kerberos authentication and indeed that so file links against Kerberos libs:

I can confirm this. I just tried disabling the workaround and reloading Icinga on three machines:

an agent with no icinga2-ido-pgsql installed
a master that is currently not maintaining the PostgreSQL connection to the IDO
a master thst is currently maintaining the PostgreSQL connection to the IDO

Of all three, only the master with the IDO connection logged the AVC event:

type=AVC msg=audit(1627897870.578:74418): avc:  denied  { search } for  pid=3189962 comm="icinga2" name="krb5" dev="dm-3" ino=524428 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1627897870.597:74419): avc:  denied  { search } for  pid=3189962 comm="icinga2" name="krb5" dev="dm-3" ino=524428 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0

So the issue is directly related to the PostgreSQL library code in the icinga2-ido-pgsql package and triggered only when the database is actually connected to.

julianbrost commented 3 years ago

Do you happen to use Kerberos anywhere? Like does a host keytab exist? Is the PostgreSQL server configured to support Kerberos/GSSAPI? Or does the client just always try to access some keytab? Also, can you figure out which file operation actually causes that error?

peteeckel commented 3 years ago

Kerberos is not in use anywhere in the network, and neither the PostgreSQL server (which is running on a different machine) nor the ido-pgsql resource is configured to even accept KRB authentication (no keyfile is configured), so it looks like the client tries GSSAPI/Kerberos authentication regardless of configuration at either end.

object IdoPgsqlConnection "ido-pgsql" {
    user = "icingaido"
    password = DBPassword
    host = "postgres.example.com"
    database = "icingaido"
    ssl_mode = "verify-full"
    ssl_ca = "/etc/pki/tls/certs/hbr-root-ca.crt.pem"
    enable_ha = true

    cleanup = {
        acknowledgements_age = 365d
        commenthistory_age   = 365d
        downtimehistory_age  = 365d
        notifications_age    = 365d
        statehistory_age     = 365d
    }
}

What I've found out so far (without debugging Icinga 2) is that libpq pulls in libgssapi_krb5 if present (which it always is on EL8 systems, as the sudo package depends on krb5-libs). As soon as there is an auth request from the database, it tries to search the keytab directory, which then triggers the SElinux violation when the program calling the libpq connectdb function is running as a confined process.

Icinga / icinga2

Icinga 2 reload logs AVC errors with SELinux on RHEL 8 #8179

Describe the bug

Expected behavior

Your Environment

============= icinga2_t ==============