Doesn't boot after pacman update

[alex-startengine]

• Device: PineTab
• Kernel Version : $(uname -r)
• UI: even minimal is broken

Steps to reproduce

download and flash latest Arch release. boot it (first boot works), do a pacman -Syu

Expected behavior

screen shows the penguin arch logo and boots the kernel

Actual behavior

Keyboard lights up, screen stays blank. installing linux-megi-5.17.0-2-aarch64

Logfiles and additional information

serial console boot log:

U-Boot SPL 2022.01-4 (Feb 17 2022 - 20:19:02 +0000)
DRAM: 2048 MiB
Trying to boot from MMC1
NOTICE:  BL31: v2.6(debug):
NOTICE:  BL31: Built : 20:19:02, Feb 17 2022
NOTICE:  BL31: Detected Allwinner A64/H64/R18 SoC (1689)
NOTICE:  BL31: Found U-Boot DTB at 0x208a4b0, model: PineTab, Development Sample
INFO:    ARM GICv2 driver initialized
INFO:    Configuring SPC Controller
INFO:    PMIC: Probing AXP803 on RSB
INFO:    PMIC: dcdc1 voltage: 3.300V
INFO:    PMIC: dcdc5 voltage: 1.200V
INFO:    PMIC: dcdc6 voltage: 1.100V
INFO:    PMIC: dldo1 voltage: 3.300V
INFO:    PMIC: dldo3 voltage: 2.800V
INFO:    PMIC: dldo4 voltage: 3.300V
INFO:    PMIC: fldo1 voltage: 1.200V
INFO:    PMIC: Enabling DC SW
INFO:    BL31: Platform setup done
INFO:    BL31: Initializing runtime services
INFO:    BL31: cortex_a53: CPU workaround for 843419 was applied
INFO:    BL31: cortex_a53: CPU workaround for 855873 was applied
INFO:    BL31: cortex_a53: CPU workaround for 1530924 was applied
INFO:    PSCI: Suspend is available via SCPI
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x4a000000
INFO:    SPSR = 0x3c9

U-Boot 2022.01-4 (Feb 17 2022 - 20:19:02 +0000) DanctNIX ARM

CPU:   Allwinner A64 (SUN50I)
Model: PineTab, Development Sample
DRAM:  2 GiB
DRAM Clock: 528 MHz
WDT:   Not starting watchdog@1c20ca0
MMC:   mmc@1c0f000: 0, mmc@1c10000: 2, mmc@1c11000: 1
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Set ram_freq : 528
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
47124480 bytes read in 1952 ms (23 MiB/s)
## Executing script at 4fc00000
Wrong image format for "source" command
SCRIPT FAILED: continuing...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@1c0f000.blk...
Card did not respond to voltage select! : -110
Scanning disk mmc@1c10000.blk...
Disk mmc@1c10000.blk not ready
Scanning disk mmc@1c11000.blk...
Found 6 disks
No EFI system partition
BootOrder not defined
EFI boot manager: Cannot load any image
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
Found U-Boot script /boot.scr
998 bytes read in 3 ms (324.2 KiB/s)
## Executing script at 4fc00000
Booting from SD
arch=arm
baudrate=115200
board=sunxi
board_name=sunxi
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_efi_bootmgr=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr;fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=fel mmc_auto pxe dhcp
bootargs=init=/init.sh rw console=tty0 console=ttyS0,115200 earlycon=uart,mmio32,0x01c28000 panic=10 consoleblank=0 loglevel=1 fbcon=rotate:1 PMOS_NO_OUTPUT_REDIRECT PMOS_FORCE_PARTITION_RESIZE pmos_boot=/dev/mmcblk0p1 pmos_root=/dev/mmcblk0p2
bootcmd=run distro_bootcmd
bootcmd_dhcp=devtype=dhcp; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_;
bootcmd_fel=if test -n ${fel_booted} && test -n ${fel_scriptaddr}; then echo '(FEL boot)'; source ${fel_scriptaddr}; fi
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc_auto=if test ${mmc_bootdev} -eq 1; then run bootcmd_mmc1; run bootcmd_mmc0; elif test ${mmc_bootdev} -eq 0; then run bootcmd_mmc0; run bootcmd_mmc1; fi
bootcmd_pxe=dhcp; if pxe get; then pxe boot; fi
bootdelay=0
bootdev=0
bootfstype=ext4
bootm_size=0xa000000
console=ttyS0,115200
cpu=armv8
devplist=1
dfu_alt_info_ram=kernel ram 0x40080000 0x1000000;fdt ram 0x4FA00000 0x100000;ramdisk ram 0x4FF00000 0x4000000
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
efi_dtb_prefixes=/ /dtb/ /dtb/current/
ethaddr=02:ba:e8:a0:c2:fb
fdt_addr_r=0x4FA00000
fdtcontroladdr=b9f3f850
fdtfile=allwinner/sun50i-a64-pinetab.dtb
fdtoverlay_addr_r=0x4FE00000
fileaddr=4fc00000
filesize=3e6
kernel_addr_r=0x40080000
kernel_comp_addr_r=0x44000000
kernel_comp_size=0xb000000
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
loadaddr=0x42000000
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
mmc_bootdev=0
partitions=name=loader1,start=8k,size=32k,uuid=${uuid_gpt_loader1};name=loader2,size=984k,uuid=${uuid_gpt_loader2};name=esp,size=128M,bootable,uuid=${uuid_gpt_esp};name=system,size=-,uuid=${uuid_gpt_system};
pinetabfdt=sun50i-a64-pinetab.dtb
pxefile_addr_r=0x4FD00000
ram_freq=528
ramdisk_addr_r=0x4FF00000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boe
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x4FC00000
serial#=92c0babae8a0c2fb
soc=sunxi
stderr=serial,vidconsole
stdin=serial
stdout=serial,vidconsole
uuid_gpt_esp=c12a7328-f81f-11d2-ba4b-00a0c93ec93b
uuid_gpt_system=b921b045-1df0-41c3-af44-4c6f280d3fae

Environment size: 4710/131068 bytes
Loading DTB
Failed to load 'sun50i-a64-pinetab.dtb'
Loading Initramfs
Failed to load 'uInitrd'
Loading Kernel
Failed to load 'vmlinuz'
Resizing FDT
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
Booting kernel
Bad Linux ARM64 Image magic!
SCRIPT FAILED: continuing...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
BootOrder not defined
EFI boot manager: Cannot load any image
No ethernet found.

[alex-startengine]

Here is the uboot log before doing the pacman update (good boot):

U-Boot 2022.01-4 (Feb 17 2022 - 20:19:02 +0000) DanctNIX ARM

CPU:   Allwinner A64 (SUN50I)
Model: PineTab, Development Sample
DRAM:  2 GiB
DRAM Clock: 528 MHz
WDT:   Not starting watchdog@1c20ca0
MMC:   mmc@1c0f000: 0, mmc@1c10000: 2, mmc@1c11000: 1
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Set ram_freq : 528
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
1491 bytes read in 2 ms (727.5 KiB/s)
## Executing script at 4fc00000
gpio: pin 98 (gpio 98) value is 1
Booting from SD
Loading kernel...
8143420 bytes read in 339 ms (22.9 MiB/s)
Uncompressing kernel...
Uncompressed size: 19873800 = 0x12F4008
Loading initramfs...
7348473 bytes read in 307 ms (22.8 MiB/s)
Loading dtb...
45100 bytes read in 7 ms (6.1 MiB/s)
Resizing FDT
Adding FTD RAM clock
memory {
        ram_freq = "528";
};
Loading user script
Failed to load '/user.scr'
No user script found
Booting...
gpio: pin 98 (gpio 98) value is 0
Moving Image from 0x40080000 to 0x40200000, end=41560000
## Flattened Device Tree blob at 4fa00000
   Booting using the fdt blob at 0x4fa00000
   Loading Ramdisk to 498fd000, end 49fff0f9 ... OK
   Loading Device Tree to 00000000498ee000, end 00000000498fcfff ... OK

Starting kernel ...

[    0.203595] lima 1c40000.gpu: dev_pm_opp_set_regulators: no regulator (mali) found: -19
[   19.572923] ov5640 3-004c: uploaded firmware didn't start, got to 0x7f, retrying...
[   19.893008] ov5640 3-004c: uploaded firmware didn't start, got to 0x7f
[   19.899655] ov5640 3-004c: ov5640_set_ctrl_focus: no autofocus firmware loaded

Arch Linux 5.16.10-1-danctnix (ttyS0)

[alex-startengine]

after manually running mkscr after doing pacman update, it boots again. there must be something blocking pacman from automatically running this?

[Danct12]

It should automatically run it, probably the command failed.

[alex-startengine]

could this have something to do with tow-boot? pinetab boots after downgrading to linux-megi-5.16.15-1

[Danct12]

We haven't started the transition from U-Boot to Tow-Boot yet.

[alex-startengine]

something i noticed during pacman update

Applying kernel sysctl settings...
Couldn't write '16' to 'kernel/sysrq', ignoring: No such file or directory

[alex-startengine]

here is the kernel serial log for any wizards who might know what's going wrong

Starting kernel ...

[    4.756595] lima 1c40000.gpu: dev_pm_opp_set_regulators: no regulator (mali) found: -19
[    9.031521] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000290
[    9.040420] Mem abort info:
[    9.043930]   ESR = 0x96000004
[    9.047445]   EC = 0x25: DABT (current EL), IL = 32 bits
[    9.052913]   SET = 0, FnV = 0
[    9.056022]   EA = 0, S1PTW = 0
[    9.059214]   FSC = 0x04: level 0 translation fault
[    9.064132] Data abort info:
[    9.067041]   ISV = 0, ISS = 0x00000004
[    9.070907]   CM = 0, WnR = 0
[    9.076249] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000042334000
[    9.083695] [0000000000000290] pgd=0000000000000000, p4d=0000000000000000
[    9.090571] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[    9.096146] Modules linked in:
[    9.099307] CPU: 1 PID: 59 Comm: kworker/u8:3 Not tainted 5.17.0-2-danctnix #1
[    9.107210] Hardware name: PineTab, Development Sample (DT)
[    9.112781] Workqueue: events_unbound deferred_probe_work_func
[    9.118626] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    9.125584] pc : of_drm_find_panel+0x78/0xd0
[    9.129856] lr : of_drm_find_panel+0x4c/0xd0
[    9.134124] sp : ffff80000966bb40
[    9.137438] x29: ffff80000966bb40 x28: 0000000000000000 x27: 0000000000000000
[    9.144575] x26: ffff80000933f640 x25: ffff00000180700d x24: ffff8000092f35a0
[    9.151712] x23: 0000000000000036 x22: 0000000000000000 x21: ffff8000092f0fb8
[    9.158849] x20: ffff00007fbe7888 x19: ffff00000944a780 x18: 0000000000000000
[    9.165986] x17: ffff800008c694f0 x16: 0000000000000000 x15: 0000000000000000
[    9.173122] x14: 0000000000000001 x13: 0000000000000002 x12: 0000000000000040
[    9.180257] x11: 0000000000000003 x10: 0101010101010101 x9 : 0000000000000000
[    9.187394] x8 : 7f7f7f7f7f7f7f7f x7 : fefefeff646c606d x6 : 1d150c11f5f38080
[    9.194530] x5 : 00007375110c151d x4 : 0000000000000000 x3 : ffff8000092f0fb8
[    9.201667] x2 : ffff8000092f0fd8 x1 : 0000000000000000 x0 : ffff00000944a7a0
[    9.208805] Call trace:
[    9.211249]  of_drm_find_panel+0x78/0xd0
[    9.215173]  sun6i_dsi_attach+0x20/0x84
[    9.219014]  mipi_dsi_attach+0x2c/0x40
[    9.222764]  ili9881c_dsi_probe+0x100/0x180
[    9.226948]  mipi_dsi_drv_probe+0x24/0x30
[    9.230958]  really_probe.part.0+0x9c/0x2fc
[    9.235143]  __driver_probe_device+0x98/0x144
[    9.239502]  driver_probe_device+0xc4/0x15c
[    9.243687]  __device_attach_driver+0xb4/0x120
[    9.248132]  bus_for_each_drv+0x78/0xd0
[    9.251969]  __device_attach+0xdc/0x184
[    9.255806]  device_initial_probe+0x14/0x20
[    9.259991]  bus_probe_device+0x9c/0xa4
[    9.263827]  deferred_probe_work_func+0x88/0xc0
[    9.268358]  process_one_work+0x1d0/0x350
[    9.269883] sunxi-mmc 1c10000.mmc: initialized, max. request size: 16384 KB, uses new timings mode
[    9.272369]  worker_thread+0x2c0/0x460
[    9.285063]  kthread+0x10c/0x110
[    9.288295]  ret_from_fork+0x10/0x20
[    9.291879] Code: d1008033 eb02003f 540001a0 f9400261 (f9414821)
[    9.297712] mmc1: new high speed SDIO card at address 0001
[    9.297968] ---[ end trace 0000000000000000 ]---
[    9.445010] mmc2: new DDR MMC card at address 0001
[    9.450784] mmcblk2: mmc2:0001 DA4064 58.2 GiB
[    9.457785]  mmcblk2: p1 p2
[    9.465747] mmcblk2boot0: mmc2:0001 DA4064 4.00 MiB
[    9.476626] mmcblk2boot1: mmc2:0001 DA4064 4.00 MiB
[   15.070972] sun4i-drm display-engine: [drm] Cannot find any crtc or sizes
[   38.744473] random: crng init done

[alex-startengine]

issue still present with updated kernel today

[Danct12]

The same oops seems to also occur with the PinePhone Pro, the latest kernel update should fix this.

[alex-startengine]

it is still broken and has the same oops and NULL pointer dereference.

More info: I manually installed manjaro kernel 5.17.5-2-MANJARO-ARM (https://gitlab.manjaro.org/manjaro-arm/packages/core/linux/-/jobs/8453) which has less patches, and patched uboot.txt to use that uncompressed kernel.

things started working.

So i think its safe to deduce that some patch is causing this issue in pinetab.

[ezra-351]

Still having this issue on my Pinephone Pro, running the mkscr file in shared storage mode doesn't seem to do anything.

[Danct12]

This issue is for the PineTab.

For the PinePhone Pro, we're aware but unable to reproduce it. Please move this discussion to #418 .

[theironrobin]

@Danct12 Can you try creating a pinetab build with this kernel config:

CONFIG_DRM_PANEL_ILITEK_ILI9881C=m
CONFIG_DRM_PANEL_SITRONIX_ST7703 is not set

source: https://salsa.debian.org/Mobian-team/devices/kernels/sunxi64-linux/-/issues/77

[aqtrans]

I just submitted a pull request which seems to fix the issue. I grabbed a patch from postmarketOS that seems to do the trick without compiling the panel as a module; I presume compiling it as a module just works around whatever timing issue this patch to the panel fixes.

I have a hastily compiled version of the linux-megi package (without the bootspash and dev-edition patches) available for testing here. My PR includes these patches again though.