Closed FiliMila closed 2 years ago
juliogonzalez
commented
3 years ago Did you follow the doc?
You did not describe how you did the upgrade, but if you used zypper dup or zypper up and that's it, then that's wrong. This is a major update.
FiliMila
commented
3 years ago Hi. Yes, I followed the doc, also because without updating susemanager before, it doesn't collect any patch/updates. I've had also applied: zypper patch --with-optional to be sure to collect everything needed; without it, some patches were conflicting between them, but now I've lost history of it; don't know if there's still any log that contains it. Please let me know. Thanks. M.
juliogonzalez
commented
3 years ago Please paste all the commands you used to perform the upgrade.
On the first look, looks as if you skipped the script that should be called to upgrade the system from Leap 15.2 to 15.3, and to upgrade the server.
FiliMila
commented
3 years ago Hi. Follows:
zypp ref
#1625058575
zypper ref
#1625058586
zypper up susemanager
#1625058611
/usr/lib/susemanager/bin/server-migrator.sh
#1625063598
/usr/lib/susemanager/bin/pg-migrate-12-to-13.sh
#1625063645
/usr/lib/susemanager/bin/pg-migrate-12-to-13.sh fast
#1625065690
journalctl -f -u uyuni-check-database.service
#1625065718
df -h
#1625065732
init 6
#1625070252
zypper ref
#1625070267
top
#1625070327
zypper ref
#1625070358
zypper list-patches
#1625070377
spacewalk-service stop
#1625070386
zypper patch
#1625070518
init 6
#1625072890
ip a
#1625072896
spacewalk-common-channels opensuse_leap15_3 opensuse_leap15_3-non-oss opensuse_leap15_3-non-oss-updates opensuse_leap15_3-updates opensuse_leap15_3-backports-update opensuse_leap15_3-sle-updates opensuse_leap15_3-uyuni-client uyuni-proxy-stable-leap-153
#1626088895
zypper patch
#1626089006
zypper refresh
#1626089057
zypper patch
#1626089085
spacewalk-common-channels opensuse_leap15_3 opensuse_leap15_3-non-oss opensuse_leap15_3-non-oss-updates opensuse_leap15_3-updates opensuse_leap15_3-backports-update opensuse_leap15_3-sle-updates opensuse_leap15_3-uyuni-client uyuni-proxy-stable-leap-153
#1626089099
spacewalk-service stop
#1626089125
zypper patch
#1626089230
init 6
#1626108410
spacewalk-service stop
#1626108422
zypper patch
#1626108888
spacewalk-common-channels opensuse_leap15_3 opensuse_leap15_3-non-oss opensuse_leap15_3-non-oss-updates opensuse_leap15_3-updates opensuse_leap15_3-backports-update opensuse_leap15_3-sle-updates opensuse_leap15_3-uyuni-client uyuni-proxy-stable-leap-153
#1626108924
zypper refresh
#1626108947
zypper patch
#1626109414
zypper list-patches
#1626109443
zypper patch --with-optional
#1626109662
init 6
#1626110119
zypper up
#1626110197
zypper list-patches
#1626110214
spacewalk-common-channels opensuse_leap15_3 opensuse_leap15_3-non-oss opensuse_leap15_3-non-oss-updates opensuse_leap15_3-updates opensuse_leap15_3-backports-update opensuse_leap15_3-sle-updates opensuse_leap15_3-uyuni-client uyuni-proxy-stable-leap-153
#1626110228
spacewalk-service stop
#1626110238
zypper patch --with-optional
#1626110246
zypper list-patches
#1626110263
zypper patch
#1626110292
spacewalk-service start
#1626110323
journalctl -f -u uyuni-check-database.service
#1626110334
spacewalk-common-channels opensuse_leap15_3 opensuse_leap15_3-non-oss opensuse_leap15_3-non-oss-updates opensuse_leap15_3-updates opensuse_leap15_3-backports-update opensuse_leap15_3-sle-updates opensuse_leap15_3-uyuni-client uyuni-proxy-stable-leap-153
#1626771281Just or let You know: spacewalk-common-channels for try to add new repo 15.3 never worked... Thanks, M.
juliogonzalez
commented
3 years ago I don't think it's what causing this, You are at least missing a reboot (https://www.uyuni-project.org/uyuni-docs/en/uyuni/upgrade/server-major-upgrade-uyuni.html) at the end of the migration.
if the services are down, of course using spacewalk-common-channels won't work.
If the reboot does not fix, please tail the logs at /var/log/tomcat and run spacewalk-services restart, then paste the output here.
Please remember to give format to the things you are pasting (I just did it for your previous comments)
FiliMila
commented
3 years ago Hi. I've rebooted many times from upgrade and it didn't solved. There's a job that never ends avoiding successful restart. I need to restart in recovery, to let systems come up succesfully.
prompt: spacewalk-service restart Shutting down spacewalk services... Done. Starting spacewalk services... Running DB schema upgrade. This may take a while. Call the following command to see progress: journalctl -f -u uyuni-check-database.service Done.
logs from journalctl: Jul 22 17:13:29 uyunis systemd[1]: Stopped Uyuni check database. Jul 22 17:13:29 uyunis systemd[1]: Starting Uyuni check database... Jul 22 17:13:30 uyunis spacewalk-startup-helper[673]: Your database schema already matches the schema package version [susemanager-schema-4.2.15-1.2.uyuni1]. Jul 22 17:13:30 uyunis spacewalk-startup-helper[673]: Schema upgrade: [susemanager-schema-4.2.15-1.2.uyuni1] -> [susemanager-schema-4.2.15-1.2.uyuni1] Jul 22 17:13:31 uyunis systemd[1]: Finished Uyuni check database.
Attaching catalina.log after restart services... I hope it may helps.
Thanks, M.
juliogonzalez
commented
3 years ago There's a job that never ends avoiding successful restart.
Well, what job exactly?
FiliMila
commented
3 years ago It's just after Reached target Remote File Systems: A start job is running for /dev/disk/by-id/sc.._different-partitions ( x min sec / no limit ) I've waited for more than 1 hour before rebooting in recovery mode.
juliogonzalez
commented
3 years ago Sadly, that could be an OS config failure, an openSUSE failure, or even a hardware failure (or network error if you are using network disks, or a lot of other different issues).
Uyuni Server was never tested with the system started in emergency mode, but it could be very well that tomcat is unable to start in emergency mode (or better said, it's trying to start, but the webapp is unable to deploy).
First thing to do is getting the system to boot up normally.
FiliMila
commented
3 years ago I can agree for application point of view, but... if it's really an hw failure, what's different - from hardware perspective - from recovery mode where it works ?
juliogonzalez
commented
3 years ago I can't tell you with only that chunk of log. You will need to use recovery mode to check what's wrong.
Hardware is one possibility, but not the only one. It could be a configuration problem somewhere at the OS, or even a bug at openSUSE itself and not Uyuni.
FiliMila
commented
3 years ago Well.. at this point, can You please suggest me a dump config I can do, if I decide to restart from scratch, without to reconfigure everything? Thanks a lot. M.
FiliMila
commented
3 years ago Just to be clear: I'm asking because I don't know if a standard backup and restore procedure (form administration guide), on a new system can regenerate the tomcat failure or not... Thanks a lot. M.
juliogonzalez
commented
3 years ago I don't think the issue you have is an Uyuni problem as nobody reported anything similar, I am unable to reproduce the issue, and it seems that your system refuses to start because of any kind of issue with the disks/partitions.
You will need to find out what exactly is going on, by reviewing the logs on maintenance mode, and the OS configuration.
If, after that problem is fixed, tomcat still doesn't start, that's when we can have a look at possible Uyuni issues.
Just to be clear: I'm asking because I don't know if a standard backup and restore procedure (form administration guide), on a new system can regenerate the tomcat failure or not...
The backup you took was, hopefully, taken before starting the update. So to restore it, you need to restore the previous OS as well, meaning you need to reinstall Leap 15.2 and whatever 2021.05
We generate snapshots for each Uyuni version. For example for 2021.05 you have: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Snapshots:/2021.05/images/
So you can follow https://www.uyuni-project.org/uyuni-docs/en/uyuni/administration/backup-restore.html, but when you are adding the Uyuni repository, add that one instead of Stable. Then follow the rest of the procedure in exactly the same way.
But before going down that road, I strongly suggest you see why the system is unable to start without emergency mode, and stops doing something with the disks.
FiliMila
commented
2 years ago Hi Julio, I've had some times to fix OS bug. During grub.cfg generation, it seems that uuid of swap partition was truncated, so it was stopping boot at first step. Now I've fixed it and - after applied patches - uyuni still hangs. Do You want try to restart from where stopped?
if the services are down, of course using
spacewalk-common-channelswon't work.
From this point of view I don't know: if I try to sync 15.2 channels, it works. It's adding 15.3 channels that fails...
Of course: main problem is always apache in down state:
_HTTP Status 404 – Not Found
Type Status Report
Description The origin server did not find a current representation for the target resource or is not willing to disclose that one exists. Apache Tomcat/9.0.36_
Ah, maybe I found something interesting... Trying to force susemanager update, received answer:
Package 'susemanager' is not installed.
Do You think may be this the problem? And if yes, how can it become uninstalled? Can You suggest me what to do now?
Thanks a lot. F.
juliogonzalez
commented
2 years ago During grub.cfg generation, it seems that uuid of swap partition was truncated, so it was stopping boot at first step.
Not a good sign, if you ask me.
Ah, maybe I found something interesting... Trying to force susemanager update, received answer: Do You think may be this the problem?
Seems to be the case, but if this package is missing, some others could be missing.
And if yes, how can it become uninstalled?
Absolutely no idea with the information we have. It could even be your RPM database got corrupted for some reason. And if that's the case, it's hard to say if something else is not corrupted as well.
In the end you should not be able to remove any of the Uyuni packages without getting a big list other packages being removed.
Are you using any kind of network storage by any chance?
Can You suggest me what to do now?
If you have a backup that can take you back to 2021.05, use it. Then make sure Uyuni is up and running, then try the upgrade again (remember: major upgrade as the OS will change from Leap 15.2 to Leap 15.3)
If you don't have a backup, try to reinstall the pattern again with sudo zypper in patterns-uyuni_server. Assuming the server migration didn't show any errors (at least you didn't tell there were any), assuming no config files were removed, and assuming nothing else is corrupted, it should work. But for me those are way too many assumptions, and personally I'd not trust such installation.
FiliMila
commented
2 years ago Hi Julio. Problem of swap partition uuid truncated is now present also onto uyuni proxy, so I'm thinking it has to be an upgrade path bug of 15.2... Said that, your suggestion worked! After reinstalling patterns-uyuni_server, system seems to be completely restored. No way to let communicate again the proxy with uyuni server, but at this point reinstalling it is the least problem (starting from point of view that 15.3 upgrade of proxy was needed too).
Summarizing: I still think that upgrade path from 2021.05 to 2021.06 has something to be checked, but now I'm onto 2021.09, so... please let me know if You think this issue has to be closed now or not.
Anyway, thanks for your help. F.
juliogonzalez
commented
2 years ago Good to hear it worked.
For the proxy, you can replace it: https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation/proxy-setup.html#_replace_a_proxy
The issue you saw could be a combination of the Leap 15.2 packages you had together with 2021.05, plus the change to Leap 15.3 with Uyuni 2021.06.
But since I was unable to reproduce it, and we don't have any other reports from the community, it could also be something very specific to your machines.
I suggest we close it, and if someone has the same issue, we can reopen it again :-)
FiliMila
commented
2 years ago ok thanks. F.
I've upgraded uyuni from 2021.05 to 2021.06; after the upgrade I get from web server:
Details about issue - patching history log:
At 19.12 tomcat has ended working as reported in the beginning.
All rhnweb*.log files ends up before reported hour.
Please let me know if I can produce more logs interesting for it. Thanks. M.