helloSystem / ISO

helloSystem Live and installation ISO
https://github.com/helloSystem/
BSD 3-Clause "New" or "Revised" License
807 stars 58 forks source link

Kernel panic during boot from live media on MacBook when SD slot is empty #505

Closed GlenHenshaw closed 1 year ago

GlenHenshaw commented 1 year ago

Describe the bug I have an iso on a USB thumb drive and an using a 2017 Intel i5 MacBook Air. The system kernel panics about six seconds into the boot process.

To Reproduce Use etcher to burn latest hello system iso file to thumb drive. Boot from thumb drive. Press the "s" key when the screen turns grey. The system kernel panics and then reboots.

Expected behavior The system should boot into the hello system GUI.

Screenshots IMG_6D31245FE518-1 Screen Shot 2023-08-04 at 8 58 46 PM

Version (please complete the following information):

Computer (please complete the following information):

Additional context Add any other context about the problem here.

probonopd commented 1 year ago

Thanks @GlenHenshaw for letting us know. Unfortunately I have never seen this before and can't deduce from looking at it what might be going wrong here. Can you boot e.g., GhostBSD on that system?

probonopd commented 1 year ago

Also, please try a different USB stick to rule out a hardware defect or incompatiblity.

GlenHenshaw commented 1 year ago

GhostBSD boots fine on this system from the same USB stick. I've tried two sticks from different vendors, both exhibit the same behavior. Also, my server machine, based on an AsRock motherboard, boot into helloSystem using the same media.

Just to be complete, I also tried burning the iso two different ways, one using the command line 'dd' command and one using the balenaEtcher macOS app. Same behavior both ways.

So this really does look like some incompatibility between the iso and the MacBook.

u20230812 commented 1 year ago

I'm seeing the same problem with both the latest release and the development version. I tried this months ago and even tried one release back and noticed the same issue. Same MacBook runs Linux just fine.

Mine is an A1502 MacBook Pro "Core i5" 2.4 13" Late 2013 2.4 GHz Core i5 (I5-4258U).

@GlenHenshaw how did you get such a clear screen grab? It goes by so fast I tried several different cameras up to 120 fps and still go a blurry frame.

Is there an easy way to enable the UART output and capture the output with a FTDI cable instead of the graphics display?

So this really does look like some incompatibility between the iso and the MacBook.

crash

macbook pro model

processor

probonopd commented 1 year ago

@u20230812 can you run GhostBSD on that machine?

We need to find out what is crashing and why it is crashing on helloSystem but not on GhostBSD.

u20230812 commented 1 year ago

@u20230812 can you run GhostBSD on that machine?

Yes, I just downloaded and checked the latest and multiuser boots right up. I tried a number of ISO with Ventoy and the Ventoy spash screen and ISO selector worked, however, trying to boot any ISO would result in a white block cursor in the upper top left part of the screen and just hang indefinitely. I had to dd the GhostBSD ISO directly to the USB drive and it booted no issues.

e399a0c193d538f09cabbdf3df397317 GhostBSD-23.06.01.iso

Let me know if there are any logs, or debug builds I can try to help isolate the issue. If it would hang instead of rebooting and we had access to the serial console that would make things a lot easier to debug and capture.

probonopd commented 1 year ago

Can you boot into single user mode (press s as soon as the screen becomes gray)?

In verbose (v) and single (s) mode the system should wait 30 seconds before rebooting. If you can get into single user mode, you can set a different time like so:

sysctl kern.panic_reboot_wait_time=300
exit
probonopd commented 1 year ago

access to the serial console

For the early boot messages? I'm not sure whether this works over serial-over-USB. According to https://forums.freebsd.org/threads/using-usb-serial-adapter-for-console-output.69759/ it looks like it might be possible.

By pressing Backspace early in the boot process you get access to the FreeBSD bootloader where you should be able to enter bootloader commands.

probonopd commented 1 year ago

Also, have you tried the older (0.7.x, 0.6.x,...) helloSystem ISOs?

u20230812 commented 1 year ago

@probonopd here are the results

probonopd commented 1 year ago

Thanks for testing. The results are interesting. We need to find out what causes the crash, but I don't know where to start yet. I don't know what is crashing there. A kernel module? Which one?

probonopd commented 1 year ago

@GlenHenshaw @u20230812 please test with and without a SD card in the slot. Possibly this issue is caused by the SD card slot being empty:

u20230812 commented 1 year ago

@probonopd the empty SD slot was exactly the problem. With a card installed there and still booting the ISO from a separate USB key it came right up and did not panic/crash/reboot.

I spent a few days building the helloSystem ISO trying to get the serial console working with a couple of USB TTL cables which never ended up working for whatever reason.

Here's some feedback on what I found to work while building the ISO. Maybe there are newer instructions, or maybe my method was incorrect, or maybe the docs could use an update...

1) Extremely easy to figure out how to get all this working based on docs available and scripts provided 2) The FreeBSD VM qcow2 filesystem is too small 3) Building the Live ISO https://hellosystem.github.io/docs/developer/building.html sha is not a package on FreeBSD 13.2 pkg install -y pkg git-lite zsync wget bash zip devel/py-xdg librsvg2 ca_root_nss sudo git zenity 4) You need to edit loader.conf before building the ISO vi + overlays/boot/boot/loader.conf uftdi_load="YES" boot_multicons="YES" boot_serial="YES" comconsole_speed="115200" console="comconsole,efi" 5) zenity package not found doing the build needs to be part of Building the Live ISO Installing furybsd-wifi-tool-0... pkg: Missing dependency 'zenity' see pkg install on item #3 5a) edit overlays/uzip/furybsd-wifi-tool/manifest and remove x11/zenity and put zenity since x11/zenity is not a package on 13.2 5b) edit overlays/uzip/furybsd-xorg-tool/manifest and remove x11/zenity and put zenity since x11/zenity is not a package on 13.2 5c) edit settings/packages.hello adding x11/zenity 6) Comment out xcb-util-cursor in settings/packages.hello 7) export DISPLAY=:0 7a) sudo su 7b) sh 8) export QT_QPA_PLATFORM=offscreen 9) QT_QPA_PLATFORM=offscreen ./build.sh hello 10) Turns out modules get removed from the builder script and don't work vi ./settings/rc.conf.common kld_list="fusefs cuse ig4 iicbus iichid utouch asmc if_urndis if_cdce if_ipheth usb ucom uftdi"

probonopd commented 1 year ago

Hi @u20230812, thanks for your valuable feedback.

For the serial console, 4. seems to be key. It might not be necessary to build special ISOs, as you can enter the FreeBSD bootloader and can type in the lines there. I will see if I can integrate this into the ISOs in an easier way.

Regarding 5., we don't use furybsd-wifi-tool anymore in helloSystem and we shouldn't need zenity anymore. Were you building another target than hello (./build.sh hello)?

But most importantly: Did you get it to work in the end?

Step 10. should not be necessary; if you are already loading uftdi at boot time (using the bootloader) then you don't need to load it again the later boot process using kld_list.

What are you trying to achieve with steps 7.-9.?

probonopd commented 1 year ago

Now, the question is: Can we fix booting with an empty SD slot using some USB Mass Storage Quirks?

probonopd commented 1 year ago

So far I coud not get serial console over USB to work; probably we need someone with more knowledge about the FreeBSD bootloader and its lua scripting capabilities.

https://github.com/helloSystem/ISO/blob/ae0880e6889ee568194b8c60135a442787b04bab/overlays/boot/boot/lua/local.lua#L82

I have opened its own ticket for that topic:

u20230812 commented 1 year ago

Now, the question is: Can we fix booting with an empty SD slot using some USB Mass Storage Quirks?

helloSystem test results on MacBook Pro with empty SD slot

  1. Stock ISO hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso hw.usb.quirk did not work still boots then crashes when the SD slot is empty
  2. Custom ISO hello-0.8.2_git-FreeBSD-13.2-amd64.iso hw.usb.quirk did not work still boots then crashes when the SD slot is empty even with usb added to kld_list vi ./settings/rc.conf.common kld_list="fusefs cuse ig4 iicbus iichid utouch asmc if_urndis if_cdce if_ipheth usb ucom uftdi"
  3. Custom ISO hello-0.8.2_git-FreeBSD-13.2-amd64.iso hw.usb.quirk did not work still boots then crashes when the SD slot is empty even with usb_quirk_load added to loader.conf vi + overlays/boot/boot/loader.conf usb_quirk_load="YES"

https://man.freebsd.org/cgi/man.cgi?query=usb_quirk&sektion=4&manpath=FreeBSD+13.2-RELEASE

probonopd commented 1 year ago

Can you reproduce the crashes with stock FreeBSD images and/or GhostBSD images?

u20230812 commented 1 year ago

Can you reproduce the crashes with stock FreeBSD images and/or GhostBSD images?

I need to amend my previous test results. It seems I blindly trusted the vendor/product IDs instead of checking them against mine before testing. All testing was done with a USB key ISO with the SD slot empty.

lsusb -s 2:3

Bus 002 Device 003: ID 05ac:8406 Apple, Inc. Internal Memory Card Reader

Setting all these allows the stock ISO hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso to boot

hw.usb.quirk.0="0x05ac 0x8406 0 0xffff UQ_MSC_NO_INQUIRY" hw.usb.quirk.1="0x05ac 0x8406 0 0xffff UQ_MSC_NO_RS_CLEAR_UA" hw.usb.quirk.2="0x05ac 0x8406 0 0xffff UQ_MSC_NO_TEST_UNIT_READY" hw.usb.quirk.3="0x05ac 0x8406 0 0xffff UQ_MSC_NO_START_STOP" hw.usb.quirk.4="0x05ac 0x8406 0 0xffff UQ_MSC_NO_INQUIRY_EVPD"

UQ_MSC_NO_INQUIRY and UQ_MSC_NO_RS_CLEAR_UA alone still ended in a panic crash. I don't have a way to connect a bus analyzer to the slot so it would be trial and error to figure out what minimal combination of the quirks above are necessary.

Setting only this one allows the stock ISO hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso to boot hw.usb.quirk.0="0x05ac 0x8406 0 0xffff UQ_MSC_IGNORE"

UQ_MSC_IGNORE will obviously ignore the whole device whereas the others just tell the driver to stop issuing discovery commands at boot for that specific device.

probonopd commented 1 year ago

Following https://github.com/helloSystem/ISO/pull/521/commits/07e39aa06ecce3ce29d058e7f21d488488fcf25f by @u20230812, please retest on hello-0.9.0_0I39-FreeBSD-13.2-amd64.iso and newer.

GlenHenshaw commented 1 year ago

hello-0.9.0_0139-FreeBSD-13.2-amd64.iso boots on my machine!

It won't recognize the Wi-Fi device, but that's a different problem.

probonopd commented 1 year ago

Added https://github.com/helloSystem/ISO/commit/b9a2d0d2ba62fb38d27df32291977c06900c0ea7.

Please retest in build 0I41 or later.

If it is still crashing, please let me know

GlenHenshaw commented 1 year ago

build 0I41 also boots on my machine