search

fedora-eln / eln

Main repository and issue tracker
10 stars 6 forks source link

TASK: guest-images doesn't boot #94

Closed ondrejbudai closed 1 month ago

ondrejbudai commented 2 years ago

I tried booting the latest Fedora-ELN-Guest-9.0-0.1654091725.x86_64.qcow2 using qemu and it failed on:

dracut-mount[409]: Warning: Can't mount root filesystem

Here's the full boot log: https://gist.github.com/ondrejbudai/ee12bfa4281a8b7df3fd58aa880bfb75

jkonecny12 commented 2 years ago

The inst.method should be inst.stage2 or inst.repo that is probably the issue here. Could you please try it with inst.stage2?

Although we should automatically replace inst.method by inst.replace so it should not be problem...

jkonecny12 commented 2 years ago

I also don't see any sign of our Dracut modules like there are not there.

jkonecny12 commented 2 years ago

Ohh, my bad. The guest images shouldn't have inst. options at all. They should be handled by could-init so the command line options are completely wrong.

sgallagher commented 2 years ago

This is the same issue I'm seeing in https://github.com/fedora-eln/eln/issues/95

For some reason, the kernel command-line isn't getting a root= argument for dracut.

Investigation is ongoing.

sgallagher commented 2 years ago

The key difference between the storage.log from ELN and one from F36 appears to be this line:

INFO:anaconda.modules.storage.bootloader.grub2:bootloader.py: used boot args: resume=UUID=a2c04c22-897a-4407-8353-e020e080c527 
INFO:anaconda.modules.storage.bootloader.grub2:bootloader.py: used boot args: rd.lvm.lv=fedora_fedora/root rhgb quiet

I've spent all day chasing this through Anaconda and Blivet, but at this point I really need to get some help from those teams. @poncovka @jkonecny12 @vojtechtrefny could one of you help me track this down?

jkonecny12 commented 2 years ago

In the log above it seems that Anaconda doesn't even start. That is correct because it shouldn't be present on guest image (cloud-init).

However, I tried to run compose from here https://odcs.fedoraproject.org/composes/ and the installed system has inst.stage2= instead of root parameter (which won't work obviously). However, I did not had a time to look on why it's happening and I have PTO from today until next Monday. However, @poncovka is the best contact here to help you, she knows the code much more than me. Sorry for not being able to help you more. :(

poncovka commented 2 years ago

The boot arguments generated by Anaconda are fine. We don't generate the root= option, I think that's done by grub2.

I was playing with Fedora-ELN-20220602.1-x86_64-dvd1.iso and, as Jirka mentioned, the boot options of the installed system are messed up. It is the cmdline of the installation environment. The Anaconda boot args are not even there.

Screenshot_vpn_test_01_live_2022-06-07_11:32:24

Interestingly, the kernelopts variable in grub.cfg contains the right data. Screenshot_vpn_test_01_live_2022-06-07_11:35:03

The /etc/default/grub file also looks file. Screenshot_vpn_test_01_live_2022-06-07_11:52:02

From the Anaconda point of view, I don't see anything unusual in the logs. Maybe the bootloader team could have some insights?

sgallagher commented 2 years ago

@frozencemetery Do you have any idea where things might be going wrong here?

frozencemetery commented 2 years ago

The boot arguments generated by Anaconda are fine. We don't generate the root= option, I think that's done by grub2.

Kind of. Arguments are complicated because systemd (kernel-install) wants to be involved, and that's what's ultimately responsible for the arguments chosen. There are too many moving parts here.

I suspect the immediate cause of this change is grubby - I made it write out the contents of GRUB_CMDLINE_LINUX to /etc/kernel/cmdline when asked to --update-kernel=ALL. My guess is that this gets called during the install process at some point, but those arguments aren't what's supposed to be preserved. (Either that or config files get edited under us.) If /etc/kernel/cmdline doesn't have a root= argument, then the cmdlines generated by kernel-install won't either. I try to pick up the one from /proc/cmdline, but if that doesn't have one, we may be out of luck...

If we can't make this work, I can revert, though that'd probably mean declaring bankruptcy on interacting with kernel-install.

frozencemetery commented 2 years ago

I guess to check that, could we see /etc/kernel/cmdline as well?

sgallagher commented 2 years ago

OK, as of the most recent compose, this doesn't seem to be happening anymore. I'm not sure what changed, unless https://github.com/rhinstaller/anaconda/pull/4158 indirectly solved this problem too.

poncovka commented 2 years ago

I guess to check that, could we see /etc/kernel/cmdline as well?

/etc/kernel/cmdline doesn't exist on the installed system.

I don't know if it can be related, but /proc/cmdline of the installed system is the cmdline of the installation environment, because we mount proc at /mnt/sysimage/proc during the installation (so this is no longer true after the reboot). However, that's not something new and it never caused any issues.

Screenshot_vpn_test_01_live_2022-06-09_12:37:07

frozencemetery commented 2 years ago

/etc/kernel/cmdline doesn't exist on the installed system.

Well, so much for that idea I guess :)