Upgrade fails from Fedora 40 to 41 container

[zwets]

Hi Jan,

I just tried to upgrade my docker-built container from fedora-40 (with commits up to 7c3410f1ef86fcd4) to fedora-41 (4df8040a89c8b80a).

Previous upgrades have always gone smoothly (I've come from about fedora-30 over the years), but this one fails on the same error that 504cabb1a507ce1a was a workaround for (and then reverted in aa6d209fcfe54cb0): the pki-tomcat.service failing to start.

I know those are very old commits, but I'm just wondering if this could be a regression introduced by Fedora 41?

[adelton]

I doubt it's something from that far ago.

We do run upgrade test from Fedora 40 to 41 and it passes -- see https://github.com/freeipa/freeipa-container/actions/runs/11658122782/job/32461097879. So it will likely require full troubleshooting (logs and stuff) to see what is specific about your installation that it did not survive the upgrade.

[zwets]

OK, let me close this for now and get back to it when I have time to dig into it (and/or upgrade to 41 becomes urgent).

[zwets]

@adelton to follow up: today trying any ipa command gave me ipa: ERROR: cannot connect to 'any of the configured servers': https://[...]/ipa/json, ..., and it turned out the certificates had expired on Oct 1st.

I bet this tripped up the upgrade. I will try and fix this with ipa-cert-fix in the containers, but wonder about the underlying cause. The renewal should be automatic, right?

[zwets]

The ipa-cert-fix command (documented here) fixed the expired certificates, in combination with ipa-getkeytab to renew /etc/krb5.keytab and /var/lib/ipa/gssproxy/http.keytab to make certmonger work again. Not sure when or how this broke, but all is normal again now.

[rcritten]

The journal should tell you what certmonger tried to do. It's unusual to have to retrieve new keytabs. Glad you are back up and running though.

[zwets]

Thanks Rob,

Could it be that these keytabs have been "broken" right from the start? This installation was precisely 2 years old when the issue emerged. If only certmonger depends on them, then the renewal process may have been broken all along?

I skimmed through the ipaserver-install.log and ipaupgrade.log, but didn't see anything suspicious there.

My reason for refreshing the keytabs was that getcert list gave

ca-error: Error setting up ccache for "host" service on client using default keytab: Preauthentication failed.

for most certificates, and kinit -kt /etc/krb5.keytab and kinit -p HTTP/$(hostname) /var/lib/ipa/gssproxy/http.keytab both failed (whereas ds.keytab and dogtag.keytab were fine).

I have the old logs in snapshots I took right before and after the fixes (ZFS is a blessing!), but cleared the log dirs for my final "go live" restart - to make grepping for future issues easier. :)

[rcritten]

The journal should tell you what certmonger tried to do. It's unusual to have to retrieve new keytabs. Glad you are back up and running though.

[zwets]

All is running well, with all certs renewed and status MONITORING, no errors logged.

The one (likely) issue I see is that on the renewal master the 3 IPA-issued certs (of the 9 total) still show the "Preauthentication failed" ca-error tag. On the replica, all 9 are fine. kinit -k works as well.

@rcritten any suggestion on what is going on and/or how to fix this?

[rcritten]

You can manually try them again with: getcert resubmit -i -vw. It may have been blocked because of the keytab. With -vw it will show the states as the certificate request goes through. Hopefully it will land in MONITORING.

[zwets]

Thanks Rob, that fixed all three.

And thanks to the whole FreeIPA/IdM team for such a solid, well-supported product!