crash loop on "Failed to start Configure IPA server upon the first start." on recent rawhide container

martinpitt commented 3 years ago

The current freeipa/freeipa-server:fedora-rawhide container image (d9f32f01c6f0 from 3 hours ago) now never finishes booting, crash-loops in ipa-server-configure-first.service, and eventually gives up. This happens on a current CentOS 7 host with docker.

[root@services ~]# docker run -it --rm --privileged --name freeipa -ti -h f0.cockpit.lan --read-only     -e IPA_SERVER_IP=10.111.112.100     -p 53:53/udp -p 53:53 -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 -p 88:88/udp -p 464:464/udp -p 123:123/udp     -v /var/lib/ipa-data:/data:Z     -v /sys/fs/cgroup:/sys/fs/cgroup:ro     freeipa/freeipa-server:fedora-rawhide     -U -p foobarfoo -a foobarfoo -n cockpit.lan -r COCKPIT.LAN --setup-dns --no-forwarders --no-ntp
systemd v248~rc2-3.fc35 running in system mode. (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.
Initializing machine ID from random generator.
Queued start job for default target Minimal target for containerized FreeIPA server.
systemd-journald.service: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling.
(This warning is only shown for the first unit using IP firewalling.)

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.
Version 4.9.2

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure the KDC to enable PKINIT

Excluded by options:
  * Configure the NTP client (chronyd)

Warning: skipping DNS resolution of host f0.cockpit.lan
Wed Mar 10 08:03:12 UTC 2021 /usr/sbin/ipa-server-configure-first 
Checking DNS domain cockpit.lan., please wait ...

The IPA Master Server will be configured with:
Hostname:       f0.cockpit.lan
IP address(es): 172.17.0.2
Domain name:    cockpit.lan
Realm name:     COCKPIT.LAN

The CA will be configured with:
Subject DN:   CN=Certificate Authority,O=COCKPIT.LAN
Subject base: O=COCKPIT.LAN
Chaining:     self-signed

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       No forwarders
Forward policy:   only
Reverse zone(s):  No reverse zone

Disabled p11-kit-proxy
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/41]: creating directory server instance
[  OK  ] Created slice system-dirsrv.slice.
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
  [2/41]: tune ldbm plugin
  [3/41]: adding default schema
  [4/41]: enabling memberof plugin
  [5/41]: enabling winsync plugin
  [6/41]: configure password logging
  [7/41]: configuring replication version plugin
  [8/41]: enabling IPA enrollment plugin
  [9/41]: configuring uniqueness plugin
  [10/41]: configuring uuid plugin
  [11/41]: configuring modrdn plugin
  [12/41]: configuring DNS plugin
  [13/41]: enabling entryUSN plugin
  [14/41]: configuring lockout plugin
  [15/41]: configuring topology plugin
  [16/41]: creating indices
  [17/41]: enabling referential integrity plugin
  [18/41]: configuring certmap.conf
  [19/41]: configure new location for managed entries
  [20/41]: configure dirsrv ccache and keytab
[**    ] A start job is running for Configure IPA server upon the first start (8s / no limit)
  [21/41]: enabling SASL mapping fallback
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
  [23/41]: adding sasl mappings to the directory
  [24/41]: adding default layout
  [25/41]: adding delegation layout
  [26/41]: creating container for managed entries
  [27/41]: configuring user private groups
  [28/41]: configuring netgroups from hostgroups
  [29/41]: creating default Sudo bind user
  [30/41]: creating default Auto Member layout
  [31/41]: adding range check plugin
  [32/41]: creating default HBAC rule allow_all
  [33/41]: adding entries for topology management
  [34/41]: initializing group membership
  [35/41]: adding master entry
  [36/41]: initializing domain level
  [37/41]: configuring Posix uid/gid generation
  [38/41]: adding replication acis
  [39/41]: activating sidgen plugin
  [40/41]: activating extdom plugin
  [41/41]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
[  OK  ] Reached target Network is Online.
         Starting Kerberos 5 KDC...
[  OK  ] Started Kerberos 5 KDC.
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
         Starting Kerberos 5 Password-changing and Administration...
[  OK  ] Started Kerberos 5 Password-changing and Administration.
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa-custodia
  [1/5]: Making sure custodia container exists
  [2/5]: Generating ipa-custodia config file
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia 
         Starting IPA Custodia Service...
[  OK  ] Started IPA Custodia Service.
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
[  OK  ] Created slice system-pki\x2dtomcatd.slice.
         Starting PKI Tomcat Server pki-tomcat...
[  OK  ] Started PKI Tomcat Server pki-tomcat.
         Stopping PKI Tomcat Server pki-tomcat...
[  OK  ] Stopped PKI Tomcat Server pki-tomcat.
         Starting PKI Tomcat Server pki-tomcat...
[  OK  ] Started PKI Tomcat Server pki-tomcat.
[ ***  ] A start job is running for Configure IPA server upon the first start (1min 45s / no limit)
         Stopping PKI Tomcat Server pki-tomcat...
[  OK  ] Stopped PKI Tomcat Server pki-tomcat.
  [3/28]: backing up CS.cfg
  [4/28]: Add ipa-pki-wait-running
  [5/28]: secure AJP connector
  [6/28]: reindex attributes
  [7/28]: exporting Dogtag certificate store pin
  [8/28]: disabling nonces
  [9/28]: set up CRL publishing
  [10/28]: enable PKIX certificate path discovery and validation
  [11/28]: authorizing RA to modify profiles
  [12/28]: authorizing RA to manage lightweight CAs
  [13/28]: Ensure lightweight CAs container exists
  [14/28]: starting certificate server instance
         Starting PKI Tomcat Server pki-tomcat...
[  OK  ] Started PKI Tomcat Server pki-tomcat.
  [15/28]: configure certmonger for renewals
         Starting Certificate monitoring and PKI enrollment...
[  OK  ] Started Certificate monitoring and PKI enrollment.
  [16/28]: requesting RA certificate from CA
[    **] A start job is running for Configure IPA server upon the first start (2min 17s / no limit)
  [17/28]: publishing the CA certificate
  [18/28]: adding RA agent as a trusted user
[    **] A start job is running for Configure IPA server upon the first start (2min 24s / no limit)
  [20/28]: Configure HTTP to proxy connections
  [21/28]: updating IPA configuration
  [22/28]: enabling CA instance
  [23/28]: migrating certificate profiles to LDAP
  [24/28]: importing IPA certificate profiles
  [25/28]: adding default CA ACL
  [26/28]: adding 'ipa' CA entry
  [27/28]: configuring certmonger renewal for lightweight CAs
  [28/28]: deploying ACME service
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
Done configuring directory server (dirsrv).
         Stopping PKI Tomcat Server pki-tomcat...
[  OK  ] Stopped PKI Tomcat Server pki-tomcat.
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
         Starting PKI Tomcat Server pki-tomcat...
[  OK  ] Started PKI Tomcat Server pki-tomcat.
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
[  OK  ] Listening on ipa-otpd socket.
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring the web interface (httpd)
  [1/21]: stopping httpd
  [2/21]: backing up ssl.conf
  [3/21]: disabling nss.conf
  [4/21]: configuring mod_ssl certificate paths
  [5/21]: setting mod_ssl protocol list
  [6/21]: configuring mod_ssl log directory
  [7/21]: disabling mod_ssl OCSP
  [8/21]: adding URL rewriting rules
  [9/21]: configuring httpd
Nothing to do for configure_httpd_wsgi_conf
  [10/21]: setting up httpd keytab
  [11/21]: configuring Gssproxy
         Starting GSSAPI Proxy Daemon...
[  OK  ] Started GSSAPI Proxy Daemon.
  [12/21]: setting up ssl
[**    ] A start job is running for Configure IPA server upon the first start (3min 11s / no limit)
[***   ] A start job is running for Configure IPA server upon the first start (3min 11s / no limit)
  [14/21]: publish CA cert
  [15/21]: clean up any existing httpd ccaches
  [16/21]: configuring SELinux for httpd
  [17/21]: create KDC proxy config
  [18/21]: enable KDC proxy
         Starting One-time temporary TLS key generation for httpd.service...
[  OK  ] Finished One-time temporary TLS key generation for httpd.service.
         Starting The Apache HTTP Server...
[  OK  ] Started The Apache HTTP Server.
  [20/21]: configuring httpd to start on boot
  [21/21]: enabling oddjobd
[  OK  ] Started privileged operations for unprivileged applications.
Done configuring the web interface (httpd).
Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
         Stopping Kerberos 5 KDC...
[  OK  ] Stopped Kerberos 5 KDC.
         Starting Kerberos 5 KDC...
[  OK  ] Started Kerberos 5 KDC.
Done configuring Kerberos KDC (krb5kdc).
         Stopping Kerberos 5 KDC...
[  OK  ] Stopped Kerberos 5 KDC.
         Starting Kerberos 5 KDC...
[  OK  ] Started Kerberos 5 KDC.
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/10]: stopping directory server
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: disabling Schema Compat
  [6/10]: starting directory server
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
  [7/10]: upgrading server
[  *** ] A start job is running for Configure IPA server upon the first start (3min 32s / no limit)
         Stopping 389 Directory Server COCKPIT-LAN....
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
  [9/10]: restoring configuration
  [10/10]: starting directory server
         Starting 389 Directory Server COCKPIT-LAN....
[  OK  ] Started 389 Directory Server COCKPIT-LAN..
Done.
Restarting the KDC
         Stopping Kerberos 5 KDC...
[  OK  ] Stopped Kerberos 5 KDC.
         Starting Kerberos 5 KDC...
[  OK  ] Started Kerberos 5 KDC.
dnssec-validation yes
Configuring DNS (named)
  [1/11]: generating rndc key file
  [2/11]: adding DNS container
  [3/11]: setting up our zone
  [4/11]: setting up our own record
  [5/11]: setting up records for other masters
  [6/11]: adding NS record to the zones
  [7/11]: setting up kerberos principal
  [8/11]: setting up named.conf
created new /etc/named.conf
created named user config '/data/etc/named/ipa-ext.conf'
created named user config '/data/etc/named/ipa-options-ext.conf'
  [9/11]: setting up server configuration
  [10/11]: configuring named to start on boot
[**    ] A start job is running for Configure IPA server upon the first start (3min 38s / no limit)
  [11/11]: changing resolv.conf to point to ourselves
Could not update DNS config: [Errno 30] Read-only file system: '/etc/resolv.conf'
Done configuring DNS (named).
         Stopping The Apache HTTP Server...
[  OK  ] Stopped The Apache HTTP Server.
         Starting One-time temporary TLS key generation for httpd.service...
[  OK  ] Finished One-time temporary TLS key generation for httpd.service.
         Starting The Apache HTTP Server...
[  OK  ] Started The Apache HTTP Server.
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
[  *** ] A start job is running for Configure IPA server upon the first start (3min 44s / no limit)
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
[  OK  ] Started IPA key daemon.
Restarting named
         Starting Generate rndc key for BIND (DNS)...
[  OK  ] Finished Generate rndc key for BIND (DNS).
         Starting Berkeley Internet Name Domain (DNS)...
[  OK  ] Started Berkeley Internet Name Domain (DNS).
[  OK  ] Reached target Host and Network Name Lookups.
Updating DNS system records
Configuring client side components
[    **] A start job is running for Configure IPA server upon the first start (3min 48s / no limit)
This program will set up IPA client.
Version 4.9.2

[     *] A start job is running for Configure IPA server upon the first start (3min 49s / no limit)
Client hostname: f0.cockpit.lan
Realm: COCKPIT.LAN
DNS Domain: cockpit.lan
IPA Server: f0.cockpit.lan
BaseDN: dc=cockpit,dc=lan

Configured sudoers in /data/etc/authselect/user-nsswitch.conf
[ ***  ] A start job is running for Configure IPA server upon the first start (3min 51s / no limit)
No valid Negotiate header in server response
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information
Configuration of client side components failed!
[FAILED] Failed to start Configure IPA server upon the first start.
See 'systemctl status ipa-server-configure-first.service' for details.
[  OK  ] Stopped target Minimal target for containerized FreeIPA server.
[  OK  ] Closed ipa-otpd socket.
         Unmounting /data...
         Unmounting /etc/hostname...
         Unmounting /etc/hosts...
         Unmounting /etc/resolv.conf...
         Unmounting /run/secrets...
         Unmounting /var/log/journal...
         Stopping Certificate monitoring and PKI enrollment...
         Stopping 389 Directory Server COCKPIT-LAN....
         Stopping GSSAPI Proxy Daemon...
         Stopping The Apache HTTP Server...
         Stopping IPA Custodia Service...
         Stopping IPA key daemon...
         Stopping Kerberos 5 Password-changing and Administration...
         Stopping Kerberos 5 KDC...
         Stopping privileged operations for unprivileged applications...
         Stopping PKI Tomcat Server pki-tomcat...
[  OK  ] Stopped Kerberos 5 Password-changing and Administration.
[  OK  ] Stopped IPA Custodia Service.
[  OK  ] Stopped Certificate monitoring and PKI enrollment.
[  OK  ] Stopped GSSAPI Proxy Daemon.
[  OK  ] Stopped privileged operations for unprivileged applications.
[  OK  ] Stopped Kerberos 5 KDC.
[FAILED] Failed unmounting /data.
[  OK  ] Unmounted /etc/hostname.
[  OK  ] Unmounted /etc/hosts.
[  OK  ] Unmounted /etc/resolv.conf.
[  OK  ] Unmounted /run/secrets.
[  OK  ] Unmounted /var/log/journal.
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Stopped IPA key daemon.
FreeIPA server configuration failed.
[  OK  ] Stopped 389 Directory Server COCKPIT-LAN..
[  OK  ] Removed slice system-dirsrv.slice.
[  OK  ] Stopped target Network is Online.
[  OK  ] Stopped PKI Tomcat Server pki-tomcat.
[  OK  ] Removed slice system-pki\x2dtomcatd.slice.
[  OK  ] Stopped The Apache HTTP Server.
[  OK  ] Stopped target Host and Network Name Lookups.
         Stopping Berkeley Internet Name Domain (DNS)...
[  OK  ] Stopped Berkeley Internet Name Domain (DNS).
[  OK  ] Stopped target System Initialization.
         Unmounting Temporary Directory (/tmp)...
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Reached target Shutdown.
[  OK  ] Unmounted Temporary Directory (/tmp).
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Reached target Final Step.
         Starting Power-Off...
[  OK  ] Finished Exit the Container.
[  OK  ] Reached target Exit the container.

martinpitt commented 3 years ago

Normally we start this without --privileged. However, that now does not work any more:

systemd v248~rc2-3.fc35 running in system mode. (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.
Failed to create /system.slice/docker-598dfa5b27446c79b3a3028a6087a54e309b9adb69cee53e2e747eb052140297.scope/init.scope control group: Operation not permitted
Failed to allocate manager object: Operation not permitted
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

But this is just fallout from the faccessat() glibc regression. That can be worked around with --security-opt=seccomp=unconfined or --privileged, but both now fail with the crash loop above.

martinpitt commented 3 years ago

Same result for freeipa/freeipa-server:fedora-33 (ef06f18112ff from 3 hours ago) and freeipa/freeipa-server:fedora-33-4.9.1 (98721900393a from 2 weeks ago).

Note that I ran each of these with an empty /var/lib/ipa-data, so it's not due to some old data.

Our previous VM image refresh with the freeipa container was on Feb 4, that still worked. That used the fedora-rawhide image ffac6c661a58 from 3 months ago. That tag is now gone on both quay and dockerhub, though.

I tested freeipa/freeipa-server:centos-7 (to match the host OS), and it fails much more quickly:

systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization other.
Detected architecture x86-64.
Set hostname to <f0.cockpit.lan>.
Initializing machine ID from random generator.
Checking DNS domain cockpit.lan, please wait ...
Wed Mar 10 08:35:37 UTC 2021 /usr/sbin/ipa-server-configure-first 

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure the KDC to enable PKINIT

Excluded by options:
  * Configure the Network Time Daemon (ntpd)

Warning: skipping DNS resolution of host f0.cockpit.lan
Checking DNS domain cockpit.lan., please wait ...

The IPA Master Server will be configured with:
Hostname:       f0.cockpit.lan
IP address(es): 172.17.0.2
Domain name:    cockpit.lan
Realm name:     COCKPIT.LAN

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       No forwarders
Forward policy:   only
Reverse zone(s):  No reverse zone

Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configure password logging
  [11/45]: configuring replication version plugin
  [12/45]: enabling IPA enrollment plugin
  [13/45]: configuring uniqueness plugin
  [14/45]: configuring uuid plugin
  [15/45]: configuring modrdn plugin
  [16/45]: configuring DNS plugin
  [17/45]: enabling entryUSN plugin
  [18/45]: configuring lockout plugin
  [19/45]: configuring topology plugin
  [20/45]: creating indices
  [21/45]: enabling referential integrity plugin
  [22/45]: configuring certmap.conf
  [23/45]: configure new location for managed entries
  [24/45]: configure dirsrv ccache
  [25/45]: enabling SASL mapping fallback
  [26/45]: restarting directory server
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
  [27/45]: adding sasl mappings to the directory
  [28/45]: adding default layout
  [29/45]: adding delegation layout
  [30/45]: creating container for managed entries
  [31/45]: configuring user private groups
  [32/45]: configuring netgroups from hostgroups
  [33/45]: creating default Sudo bind user
  [34/45]: creating default Auto Member layout
  [35/45]: adding range check plugin
  [36/45]: creating default HBAC rule allow_all
  [37/45]: adding entries for topology management
  [38/45]: initializing group membership
  [39/45]: adding master entry
  [40/45]: initializing domain level
  [41/45]: configuring Posix uid/gid generation
  [42/45]: adding replication acis
  [43/45]: activating sidgen plugin
  [44/45]: activating extdom plugin
  [45/45]: configuring directory to start on boot
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Done configuring kadmin.
Configuring ipa-custodia
  [1/5]: Making sure custodia container exists
  [2/5]: Generating ipa-custodia config file
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia 
  [5/5]: configuring ipa-custodia to start on boot
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/30]: configuring certificate server instance
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
  [2/30]: secure AJP connector
  [3/30]: reindex attributes
  [4/30]: exporting Dogtag certificate store pin
  [5/30]: stopping certificate server instance to update CS.cfg
  [6/30]: backing up CS.cfg
  [7/30]: disabling nonces
  [8/30]: set up CRL publishing
  [9/30]: enable PKIX certificate path discovery and validation
  [10/30]: starting certificate server instance
  [11/30]: configure certmonger for renewals
Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists
  [12/30]: requesting RA certificate from CA
xargs: /usr/sbin/ipa-server-install: terminated by signal 9
FreeIPA server configuration failed.

For the record: I checked the history, and it seems the only reason to use the :fedora-rawhide tag was that there was/is no :latest tag any more, and you recommended us to use rawhide instead (that would also give us the latest version to test against, which spots errors earlier).

adelton commented 3 years ago

Can you try tests/run-partial-tests.sh Dockerfile.fedora-rawhide on that box / setup to see if that passes? You might need to patch it with that --security-opt=seccomp=unconfined at https://github.com/freeipa/freeipa-container/blob/master/tests/run-partial-tests.sh#L28.

I don't see things failing on my Fedora 33 (even if I see the glibc/seccomp issue here) and it will take me some time to setup a RHEL 7 box to try to reproduce.

adelton commented 3 years ago

Yes, about the tags -- people complained that IPA gets upgraded to latest version when they used :latest (and when that happens after long time, the upgrade might fail because it's upgrade both across Fedora versions and across FreeIPA versions).

So we started to tag with specific FreeIPA versions as well in https://hub.docker.com/r/freeipa/freeipa-server/tags?page=1&ordering=last_updated and https://quay.io/repository/freeipa/freeipa-server?tab=tags. The rawhide image hasn't been built for a while, exactly because I did not want to break people's setups with that glibc issue ... but then I figured there is no point waiting if that change is there to stay and people need to workaround for example with --security-opt=seccomp=unconfined.

As for running these images on RHEL 7 hosts, it's mostly outside of my capacity to test there, I'm generally happy when things pass on my Fedoras and on GitHub Actions' Ubuntus (we no longer test on Travis CI because we were not approved for the OSS credits (yet?)).

Could you try the same on RHEL 8 machine, likely with podman? The tests that we have are the tests/run-partial-tests.sh Dockerfile.<version> which also tests basic systemd operation in the container before even attempting to configure FreeIPA, and tests/run-master-and-replica.sh <image> which is primarily for using the "real" image, testing master and replica configuration.

martinpitt commented 3 years ago

Simplest way to reproduce is with the CentOS 7 cloud image:

curl -L -O https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2.xz
xz -d CentOS-7-x86_64-GenericCloud.qcow2.xz
# nothing fancy, just admin:foobar and root:foobar
curl -L -O https://github.com/cockpit-project/bots/raw/master/machine/cloud-init.iso
qemu-system-x86_64 -enable-kvm -nographic -m 2048 -drive file=CentOS-7-x86_64-GenericCloud.qcow2,if=virtio -snapshot -cdrom cloud-init.iso -net nic,model=virtio -net user,hostfwd=tcp::2201-:22

Note: you can also log in on the VT, but ssh login with `` is a bit more comfortable:

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -p 2201 root@localhost

(Password "foobar").

Then:

yum install -y docker
systemctl start docker
setsebool -P container_manage_cgroup 1
# see https://github.com/freeipa/freeipa-container/issues/348
rm /usr/libexec/oci/hooks.d/oci-systemd-hook
mkdir /var/lib/ipa-data
docker run -it --rm --privileged --name freeipa -ti -h f0.cockpit.lan --read-only     -e IPA_SERVER_IP=10.111.112.100     -p 53:53/udp -p 53:53 -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 -p 88:88/udp -p 464:464/udp -p 123:123/udp     -v /var/lib/ipa-data:/data:Z     -v /sys/fs/cgroup:/sys/fs/cgroup:ro     freeipa/freeipa-server:fedora-rawhide     -U -p foobarfoo -a foobarfoo -n cockpit.lan -r COCKPIT.LAN --setup-dns --no-forwarders --no-ntp

I'll try to move our image to CentOS 8 stream. We need to do that at some point anyway. I'll report back here how it works on CentOS 8.

martinpitt commented 3 years ago

Indeed the current container works on CentOS 8 stream. (Unfortunately https://github.com/candlepin/ansible-role-candlepin is still not ported to RHEL/CentOS 8, so we are kind of stuck there, but I'll see what we can do there)

adelton commented 3 years ago

@martinpitt, I assume you have found reasonably stable setup for your use case. Is there anything else we should investigate or do as part of this issue?

martinpitt commented 3 years ago

@adelton : Yes, I applied a big hammer to candlepin and moved the whole host to Fedora CoreOS. So this does not block us any more. I suppose you can close this if you don't want to support running on RHEL/CentOS 7 any more.

Yamakasi commented 3 years ago

I seem to run into this as well, I'm not sure but I think I do.

With normal command I end up with:

Adding [10.1.0.3 ipa-01.foo.tld] to your /etc/hosts file
[Errno 30] Read-only file system: '/etc/hosts'
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...

Any comments ?

adelton commented 3 years ago

If you see the error about /etc/hosts, it's a different problem than originally reported by Martin which was about running latest systemd in container on CentOS 7. It's incidently one that I filed earlier today as https://pagure.io/freeipa/issue/8888 against FreeIPA.

I assume that you've run the container as read-only and with --hostname=ipa-01.foo.tld --ip-address=10.1.0.3 or similar parameters. The ipa-server-install installer is eager to add those value to /etc/hosts. One possibility to avoid that is to use something like --add-host ipa-01.foo.tld:10.1.0.3 option to docker run or podman run to make the installer happy to find the records already there.

huww98 commented 3 years ago

I run into this "No valid Negotiate header in server response" issue today after upgrade from fedora-33-4.9.2 image to fedora-34-4.9.6.

I first hit "Failed to allocate manager object.", then I read this thread, and added --privileged. Then I got "No valid Negotiate header in server response" for any ipa command. However --security-opt=seccomp=unconfined works in my case. See this mail thread for more.

Before I realize that, I spent many hours digging into the "No valid Negotiate header in server response" issue. Finally I found out that it is because apache is using a private /tmp dir, and we symlink /var/lib/gssproxy to /tmp, so apache cannot contact gssproxy.

It works with a systemd unit override:

# /data/etc/systemd/system/httpd.service.d/override.conf
[Service]
PrivateTmp=false

I guess we should add this into the container image? But not sure where to add it.

We don't have any issue when we don't use --privileged may because in that case, systemd in container does not have the privilege to create the private tmp, so it just ignores this.

adelton commented 2 years ago

I'm afraid running on CentOS / RHEL 7 hosts is no longer something we are able to reasonably support, especially with the new cgroups defaults.

freeipa / freeipa-container

crash loop on "Failed to start Configure IPA server upon the first start." on recent rawhide container #383