RFE: Fedora 28 for RockPro64

[kfiresmith]

I'd love to buy one or more of the 4GB RAM RockPro64s but we have a hard requirement for Fedora in my location. Are you considering adding a Fedora build to your linux-build repo?

Thanks!

[Xake]

To be honest, a Fedora-build is not really needed. What is needed is to be able to flash uboot to SPI, and then be able to boot any Fedora aarch64 written to a SD-card or eMMC or USB from that uboot-instance. I did this for a while on rock64 and it works nice. This worked a bit during between 0.6.0 and 0.6.5, however with "0.6.6: Use Rockchip's u-boot tree" this broke again.

@ayufan Is it possible to rebase the efi-parts of the current uboot-tree against mainline, since the rockchip-uboot-tree is based on a too old version of uboot? Or is it possible for you to maybe make spi-images based on both mainline and rockchips uboot-tree?

[BioSehnsucht]

I would also be interested, in our case for Rock64. I'd be happy to take @Xake 's solution of booting from SPI to a vanilla Fedora image, but it sounds like that doesn't work any more ? Or has this been fixed since then?

[BioSehnsucht]

I've tried it. Current SPI flash from ayufan-rock64/linux-u-boot releases seems to work fine to boot things that already worked.

I can't boot Fedora though. Might just be a Fedora thing, I was hoping this would work around the lack of proper Fedora support for the Rock64.

When booting AArch64 :

Scanning usb 0:1...
Found EFI removable media binary efi/boot/bootaa64.efi
reading efi/boot/bootaa64.efi
868376 bytes read in 71 ms (11.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
## Starting EFI application at 02000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 9
Scanning disk rksdmmc@ff520000.blk...
Card did not respond to voltage select!
mmc_init: -95, time 9
Scanning disk rksdmmc@ff500000.blk...
Scanning disk usb_mass_storage.lun0...
Found 3 disks
Something has gone seriously wrong: Device Error
Shim was unable to measure state into the TPM

When booting armhfp :

Scanning usb 0:2...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
510 bytes read in 106 ms (3.9 KiB/s)
Ignoring unknown command: ui
Ignoring malformed menu command:  autoboot
Ignoring malformed menu command:  hidden
Ignoring unknown command: totaltimeout
Fedora-Server-armhfp-28-1.1 Boot Options.
1:      Fedora-Server-armhfp-28-1.1 (4.16.3-301.fc28.armv7hl)
Enter choice: 1:        Fedora-Server-armhfp-28-1.1 (4.16.3-301.fc28.armv7hl)
Retrieving file: /initramfs-4.16.3-301.fc28.armv7hl.img
56591459 bytes read in 2684 ms (20.1 MiB/s)
Retrieving file: /vmlinuz-4.16.3-301.fc28.armv7hl
6672896 bytes read in 396 ms (16.1 MiB/s)
append: ro root=UUID=7ea4a72f-1f7f-42c0-a0d6-f75138f2419b
Retrieving file: /dtb-4.16.3-301.fc28.armv7hl/rockchip/rk3328-rock64.dtb
** File not found /dtb-4.16.3-301.fc28.armv7hl/rockchip/rk3328-rock64.dtb **
Skipping Fedora-Server-armhfp-28-1.1 (4.16.3-301.fc28.armv7hl) for failure retrieving fdt

[Xake]

The error you get from aarch64 about the shim is because missing info from u-boot to the shim/grub. You can bypass this shim by in u-boot setenv to load grub directly and not the EFI-shim. However this will have problem as well, as the uboot these images are based on are over 1 year old and misses many commits done in upstream uboot againt the EFI-implementation IIRC. I will try to verify this again this weekend.

[Xake]

So we have two problems to make Fedora "just boot". The first is that the current implementation of u-boot has not all the "signed"-variables set that makes the shim choke. This is easily worked around by using setenv to have u-boot loading efi/fedora/grubaa64.efi directly instead of bootaa64.efi. A saveenv away from "just working boot".

The second one is more problematic. Since the used version of u-boot misses commits like the following grub will not find its config file, and halt at a prompt where you have to point out the file youself. This one was the main working thing reverting for me when going back to the rockchips u-boot instead of using the upstream u-boot. However this version of the rockchip u-boot has diverged so much from mainline u-boot that backporting all the needed commits is a bit challangeing.

commit 884bcf6f65c414dce3b3d2a91e2c9eba0e5e08f8 Author: Rob Clark robdclark@gmail.com Date: Wed Sep 13 18:05:31 2017 -0400

efi_loader: use proper device-paths for partitions

[ayufan]

@Xake: is there anything that I should pick? I generally think that soon (tm) will switch to mainline u-boot.

[Xake]

@ayufan If you are going for mainline u-boot soon I would say: do not bother with this, and focus on that instead. The commit commented I know is needed because of the commit message, however for me grub still fails to find the config file with this commit, and I am unsure of what more commits is needed, or if it just is my tests that are botched for other reasons.:-P

[ayufan]

What happens? Can you list me what is part table? What is content of boot part?

[Xake]

My setup consists of a microsd-card and one usb-device. The usb-device contains the image from https://dl.fedoraproject.org/pub/fedora-secondary/releases/test/29_Beta/Spins/aarch64/images/ written following the instructions in https://fedoraproject.org/wiki/Architectures/AArch64/F28/Installation#SBC_Disk_Images but using "taget" none instead of rpi3. This will create a USB-device with 3 partitions:

Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  200MB   199MB   primary  fat16        boot
 2      200MB   1274MB  1074MB  primary  ext4
 3      1274MB  5368MB  4094MB  primary  ext4

The first one is a pretty standard efi-partition.

The microsd-card is created by:

$ ./dev-make BOARD_TARGET=rock64 u-boot-build
$ sudo dd if=tmp/u-boot-rock64/rksd_loader.img of=<sdcard> seek=64

Trying to boot this results in:

reading efi/boot/bootaa64.efi
858216 bytes read in 57 ms (14.4 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
## Starting EFI application at 02000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 9
Scanning disk rksdmmc@ff520000.blk...
Scanning disk rksdmmc@ff500000.blk...
Scanning disk usb_mass_storage.lun0...
Found 3 disks
Could not verify MokList: Device Error
Could not verify MokListX: Device Error
Could not verify MokSBState: Device Error
Could not verify MokDBState: Device Error
Something has gone seriously wrong: import_mok_state() failed
: Device Error

The workaround for this is holding enter while booting and running the following commands:

setenv boot_efi_binary 'load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/fedora/grubaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi'
setenv scan_dev_for_efi 'setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/fedora/grubaa64.efi; then echo Found EFI removable media binary efi/fedora/grubaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile'
saveenv
run bootcmd

Now Grub will boot with the following:

Found EFI removable media binary efi/fedora/grubaa64.efi
reading efi/fedora/grubaa64.efi
1881848 bytes read in 83 ms (21.6 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
## Starting EFI application at 02000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 9
Scanning disk rksdmmc@ff520000.blk...
Scanning disk rksdmmc@ff500000.blk...
Scanning disk usb_mass_storage.lun0...
Found 3 disks
Speed: 1000, full duplex
      Minimal BASH-like line editing is supported. For the first word,    
      TAB lists possible command completions. Anywhere else TAB lists   
      possible device or file completions.                              

grub> 

Why it does not load the configfile is because it tries to run it from the wrong device (root should in my case be "hd1,msdos1" and prefix should be "(hd1,msdos1)/EFI/fedora"), so it misses the partitions:

grub>set                                                                        
?=0                                                                             
btrfs_relative_path=                                                            
btrfs_subvol=                                                                   
btrfs_subvolid=                                                                 
color_highlight=black/light-gray                                                
color_normal=light-gray/black                                                   
feature_200_final=y                                                             
feature_all_video_module=y                                                      
feature_chainloader_bpb=y                                                       
feature_default_font_path=y                                                     
feature_menuentry_id=y                                                          
feature_menuentry_options=y                                                     
feature_nativedisk_cmd=y                                                        
feature_ntldr=y                                                                 
feature_platform_search_hint=y                                                  
feature_timeout_style=y                                                         
fw_path=(hd1)/efi/fedora                                                        
grub_cpu=arm64                                                                  
grub_netfs_type=grub                                                            
grub_platform=efi                                                               
lang=                                                                           
locale_dir=                                                                     
net_default_ip=(null)                                                           
net_default_mac=(null)                                                          
net_default_server=                                                             
pager=                                                                          
prefix=(hd1)/EFI/fedora                                                         
pxe_default_server=                                                             
root=hd1                                                                        
secondary_locale_dir=  

Applying only 884bcf6f65c414dce3b3d2a91e2c9eba0e5e08f8 will not compile, you need at least the following commits uptil:

396d0481011fc84f36dde1cb2568dceaba858464
f015d301f14936e6c4a5e2434076aef8cac18022
bd46bb32fb76b7824815d8f0b739b46d8e19233c
536bcb5a905dca72d2b0ac6d2f0caf49a9bc98fb

The problem is that this will leave your prefix and root in grub empty. I have yet to find what the missing commit is.

You can still boot from grub with: grub>configfile (hd1,msdos1)/efi/fedora/grub.cfg

[Xake]

I just got it working.

I applied the following commits from upstream (not sure if all really are needed... But applies cleanly):

396d0481011fc84f36dde1cb2568dceaba858464
f015d301f14936e6c4a5e2434076aef8cac18022
bd46bb32fb76b7824815d8f0b739b46d8e19233c
536bcb5a905dca72d2b0ac6d2f0caf49a9bc98fb
02cabc82a86f8a6851b3ac33330c697745a1649f
e03ec771c2a7036d3ef9887c156f066814e534e0
c3a0e4518ffde0a3c91c893f54c6181982a3e9e0
0c14c991dda3ebe511f21b65f10a266f9d902a60

ran:

$ ./dev-make BOARD_TARGET=rock64 u-boot-build
$ sudo dd if=tmp/u-boot-rock64/rksd_loader.img of=<sdcard> seek=64

During boot I applied the the following WA to the u-boot-env:

setenv boot_efi_binary 'load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/fedora/grubaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi'
setenv load_efi_dtb 'load ${devtype} ${devnum}:${dtb_devp} ${fdt_addr_r} ${prefix}${efi_fdtfile} && run boot_efi_binary'
setenv scan_dev_for_dtb 'setenv efi_fdtfile ${fdtfile}; part list ${devtype} ${devnum} dtb_devplist;env exists dtb_devplist || setenv dtb_devplist ${distro_bootpart};for dtb_devp in ${dtb_devplist}; do for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${dtb_devp} ${prefix}${efi_fdtfile}; then echo Found DTB ${devtype} ${devnum}:${dtb_devp} ${prefix}${efi_fdtfile};run load_efi_dtb; fi;done;done;run boot_efi_binary'
setenv scan_dev_for_efi 'if test -e ${devtype} ${devnum}:${distro_bootpart} efi/fedora/grubaa64.efi; then echo Found EFI removable media binary efi/fedora/grubaa64.efi; run scan_dev_for_dtb; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile'
saveenv
run bootcmd

This work around is for the following fedora downstream patches: https://src.fedoraproject.org/rpms/uboot-tools/raw/master/f/uefi-use-Fedora-specific-path-name.patch (this one changes u-boot to run grub instead of the shim) https://src.fedoraproject.org/rpms/uboot-tools/raw/master/f/uefi-distro-load-FDT-from-any-partition-on-boot-device.patch (this one makes u-boot load dtbs from other partitions then the efi patition, like for example the "/boot"-partition)

After this the kernel boot nicely, and also has no problems handling a reboot on its own.

[BioSehnsucht]

Is it possible to somehow save all this setenv stuff into the u-boot ?

Is there anything materially different in using SPI to boot u-boot with only sd-card or USB present, and fedora image in either of those? (would having only "one drive" be helpful in reducing the amount of setenv to get it working?)

[Xake]

Is it possible to somehow save all this setenv stuff into the u-boot ?

How do you mean? If you read the WA it has "saveenv" which essentially makes the WA permanent until you clear out the saved environment.

Is there anything materially different in using SPI to boot u-boot with only sd-card or USB present, and fedora image in either of those? (would having only "one drive" be helpful in reducing the amount of setenv to get it working?)

Nope. The reason the env needs to be altered is to make the the setup Fedora has chosen work. It has nothing to do with from where u-boot and the alike is executed. You can remove some parts by manually copy the dtb-directory in "/boot" to "/boot/efi/dtb", however the environment still will by default try to run the shim, and that will fail. So since you need to change the environment, then why not do the full change? And as I said, it is saved, and remains between poweroffs.

[Xake]

@ayufan Is it possible to get those upstream u-boot commits into the u-boot repo? Even if the WA stills is required for booting Fedora, it does not feel as such a hurdle as also patching and compiling u-boot from your tree.

[ayufan]

Can you send PR?

On Thu, 25 Oct 2018 at 20:54, Peter Hjalmarsson notifications@github.com wrote:

@ayufan https://github.com/ayufan Is it possible to get those upstream u-boot commits into the u-boot repo? Even if the WA stills is required for booting Fedora, it does not feel as such a hurdle as also patching and compiling u-boot from your tree.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ayufan-rock64/linux-build/issues/222#issuecomment-433165936, or mute the thread https://github.com/notifications/unsubscribe-auth/ACTpQbtUzPkCSTsZwfXLTSuHt3omDhG3ks5uogj8gaJpZM4UqA83 .

[Xake]

Done.

[bugzy]

@Xake Any chance you can give the steps as to how you got F28 on your Rock64? I did the following, but could not get anything to show up on the display:

1. Flash/erase SPI with U-Boot
2. Download Fedora image from: https://dl.fedoraproject.org/pub/fedora-secondary/releases/29/Spins/aarch64/images/
3. Install fedora-arm-installer: dnf install fedora-arm-installer
4. Write Image to media: arm-image-installer --image=Fedora-Minimal-29-1.2.aarch64.raw.xz --target=none --media=/dev/sdc --selinux=OFF --resizefs
5. Reset Rock64 and Insert New SD into it.

Results were nothing! Any thoughts on this will be appreciated.

[Xake]

Sorry, I am only using a serial console, have not tried if u-boot outputs to display. Because of that I cannot say if you can change the u-boot env from within u-boot if you do not have a serial console.

[Linutux42]

Hello @ayufan ,

I just tried what @bugzy suggested but with the latest fedora 30 image and got the following output.

U-Boot SPL board init

U-Boot SPL 2017.09-rockchip-ayufan-1045-g9922d32c04 (Mar 21 2019 - 17:27:29)
booted from SPI flash
Trying to boot from SPI
NOTICE:  BL31: v1.3(debug):370ab80
NOTICE:  BL31: Built : 09:23:41, Mar  4 2019
NOTICE:  BL31: Rockchip release version: v1.1
INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    plat_rockchip_pmu_init(1181): pd status 3e
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9

U-Boot 2017.09-rockchip-ayufan-1045-g9922d32c04 (Mar 21 2019 - 17:27:35 +0000)

Model: Pine64 RockPro64
DRAM:  3.9 GiB
DCDC_REG1@vdd_center: ; enabling
DCDC_REG2@vdd_cpu_l: ; enabling
DCDC_REG3@vcc_ddr: ; enabling (ret: -38)
DCDC_REG4@vcc_1v8: set 1800000 uV; enabling
LDO_REG1@vcc1v8_dvp: set 1800000 uV; enabling
LDO_REG2@vcc3v0_touch: set 3000000 uV; enabling
LDO_REG3@vcc1v8_pmu: set 1800000 uV; enabling
LDO_REG4@vcc_sd: set 3300000 uV; enabling
LDO_REG5@vcca3v0_codec: set 3000000 uV; enabling
LDO_REG6@vcc_1v5: set 1500000 uV; enabling
LDO_REG7@vcca1v8_codec: set 1800000 uV; enabling
LDO_REG8@vcc_3v0: set 3000000 uV; enabling
SWITCH_REG1@vcc3v3_s3: ; enabling (ret: -38)
SWITCH_REG2@vcc3v3_s0: ; enabling (ret: -38)
vcc1v8-s0@vcc1v8_s0: set 1800000 uV; enabling (ret: -38)
dc-12v@dc_12v: set 12000000 uV; enabling (ret: -38)
vcc-sys@vcc_sys: set 5000000 uV; enabling (ret: -38)
vcc3v3-sys@vcc3v3_sys: set 3300000 uV; enabling (ret: -38)
vcc-phy-regulator@vcc_phy: ; enabling (ret: -38)
vdd-log@vdd_log: ; enabling (ret: -38)
MMC:   sdhci@fe330000: 0, dwmmc@fe320000: 1
SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

In:    serial@ff1a0000
Out:   serial@ff1a0000
Err:   serial@ff1a0000
Model: Pine64 RockPro64
Net:   eth0: ethernet@fe300000
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found EFI removable media binary efi/boot/bootaa64.efi
reading efi/boot/bootaa64.efi
858216 bytes read in 59 ms (13.9 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
## Starting EFI application at 02000000 ...
Scanning disk sdhci@fe330000.blk...
Card did not respond to voltage select!
mmc_init: -95, time 9
Scanning disk dwmmc@fe320000.blk...
MMC: block number 0x1 exceeds max(0x0)
MMC: block number 0x1 exceeds max(0x0)
MMC: block number 0x44 exceeds max(0x0)
Found 2 disks
Could not verify MokList: Device Error
Could not verify MokListX: Device Error
Could not verify MokSBState: Device Error
Could not verify MokDBState: Device Error
Something has gone seriously wrong: import_mok_state() failed
: Device Error

If you have any idea i'm all ears :)

[Linutux42]

@Xake , I just tried to build your repo linux-u-boot with the added commits and copy it to my fedora setuped emmc. Unfortunately i have the same error as above. Also, holding enter while booting did not allow me to writ anything. Did I missed a step ?

[Xake]

@linutux42, my repo was only a "PR"-repo for ayufan. For Fedora you still need the WA to call /fedora/grub.efi instead. The WA is documented earlier. Or you can apply the fedora-patch mentioned which does the same but requires building uboot yourself. The error message suggests that this is your problem.

[Xake]

@linutux42

Realized you have a access to uboot. Press and hold enter while rebooting rock64. When you get the command prompt run the following :

setenv boot_efi_binary 'load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/fedora/grubaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi'
setenv load_efi_dtb 'load ${devtype} ${devnum}:${dtb_devp} ${fdt_addr_r} ${prefix}${efi_fdtfile} && run boot_efi_binary'
setenv scan_dev_for_dtb 'setenv efi_fdtfile ${fdtfile}; part list ${devtype} ${devnum} dtb_devplist;env exists dtb_devplist || setenv dtb_devplist ${distro_bootpart};for dtb_devp in ${dtb_devplist}; do for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${dtb_devp} ${prefix}${efi_fdtfile}; then echo Found DTB ${devtype} ${devnum}:${dtb_devp} ${prefix}${efi_fdtfile};run load_efi_dtb; fi;done;done;run boot_efi_binary'
setenv scan_dev_for_efi 'if test -e ${devtype} ${devnum}:${distro_bootpart} efi/fedora/grubaa64.efi; then echo Found EFI removable media binary efi/fedora/grubaa64.efi; run scan_dev_for_dtb; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile'
saveenv
run bootcmd

[Linutux42]

@Xake Thank you for your answer, I will definitely test that out this evening.

Though i have a question, i tried to hold enter key while booting but on the serial console, Does that work ? Or should i plug a usb keyboard directly to the board maybe ?

[Xake]

I myself only has my rock64 connected over serial, that is what I have tried and verified worked.

During bootup U-Boot has a short window where any key pressed will halt the boot. This is so short, that the only way to not miss it is more or less keep the key pressed during bootup until the prompt shows up.

[Linutux42]

Hello @Xake,

Well, i have a rockpro64 and whatever i try as input during boot, it never gives me the possibility to write something and i have the same output everytime.

I can definitely see a line saying "Hit any key to stop autoboot:" but it's not working. I tried with minicom and screen. If you have any tips :)

Thanks

[Xake]

Remove the fedora boot media and network cable. After uboot has tried and failed booting over pxe it will go to prompt. If you cannot input by then, verify that your serial connector actually works.

[Linutux42]

Hey @Xake,

Quick update, I interrupted succesfully the u-boot sequence and copy pasted what you've given me and i confirm that it works great ! My issue was that my TX cable was plugged on my USB but not on my board ... Now I need to fix the boot issues i have but that's on the distribution level :)

Thank you Xake

[nullr0ute]

So for reference this should work, with shim and all the standard upstream U-Boot pieces for some time with Fedora.