FSBL fails during QSPI boot

I have a custom board with a Zynq 7010 (XC7Z010-1CLG225I). There is a 16MiB NOR Flash (ISSI IS25WP128) connected to QSPI controller.

When powered on with boot mode set to QSPI, the FSBL will run and output to uart1 (our debug console):

Xilinx First Stage Boot Loader 
Release 2020.1  Apr 14 2021-07:23:49
Devcfg driver initialized 
Silicon Version 3.1
Boot mode is QSPI
Single Flash Information
FlashID=0x9D 0x60 0x18
ISSI 128M Bits
QSPI is in single flash connection
QSPI is in 4-bit mode
QSPI Init Done 
Flash Base Address: 0xFC000000
Reboot status register: 0x60400000
Multiboot Register: 0x0000C000
Image Start Address: 0x00000000
Partition Header Offset:0x88888888

Nothing happens after that.

I am using Petalinux v2020.1 and Vivado v2020.1. Our intention is to use regular non-secure boot flow and we do not want encryption or secure boot. This is the BIF file being used:

~/avnet-petalinux/projects/minized_2020_1$ cat build/bootgen.bif 
the_ROM_image:
{
    [bootloader] /tmp/tmp.xotPM1rD9F/zynq_fsbl.elf
    /tmp/tmp.xotPM1rD9F/system.bit
    /tmp/tmp.xotPM1rD9F/u-boot.elf
    [load=0x00100000] /tmp/tmp.xotPM1rD9F/system.dtb
    [offset=0x300000, partition_owner=uboot] /tmp/tmp.xotPM1rD9F/image_INITRD_MINIMAL.ub
    [offset=0xFC0000] /tmp/tmp.xotPM1rD9F/avnet_qspi.scr
}

This is programmed onto the QSPI with:

bss@bss-VirtualBox:~/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1$ cat boot_qspi_INITRD.sh
#!/bin/bash

# This script will generate a BOOT.BIN file and program the qspi
# This BOOT.BIN file will contain uboot, a kernel with INITRD and a boot.scr

# Stop the script whenever we had an error (non-zero returning function)
set -e

petalinux-package --boot --fsbl ./images/linux/zynq_fsbl.elf --fpga ./images/linux/system.bit --uboot --kernel ./image_INITRD_MINIMAL.ub -o BOOT_LINUX_QSPI.BIN --force --boot-device flash --add ./images/linux/avnet-boot/avnet_qspi.scr --offset 0xFC0000

program_flash -f ./BOOT_LINUX_QSPI.BIN -offset 0 -flash_type qspi-x4-single -fsbl ./images/linux/zynq_fsbl.elf

This is the output:

~/avnet_petalinux/projects/minized_2020_1$ bash ./boot_qspi_INITRD.sh 
INFO: sourcing build tools
INFO: Getting system flash information...
INFO: File in BOOT BIN: "/home/bss/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1/images/linux/zynq_fsbl.elf"
INFO: File in BOOT BIN: "/home/bss/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1/images/linux/system.bit"
INFO: File in BOOT BIN: "/home/bss/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1/images/linux/u-boot.elf"
INFO: File in BOOT BIN: "/home/bss/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1/images/linux/system.dtb"
INFO: File in BOOT BIN: "/home/bss/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1/image_INITRD_MINIMAL.ub"
INFO: File in BOOT BIN: "/home/bss/Projects/DEVO/DEVO_Petalinux/projects/minized_2020_1/images/linux/avnet-boot/avnet_qspi.scr"
INFO: Generating Zynq binary package BOOT_LINUX_QSPI.BIN...

****** Xilinx Bootgen v2020.1
  **** Build date : May 26 2020-14:07:15
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.

[INFO]   : Bootimage generated successfully

INFO: Binary is ready.
WARNING: Unable to access the TFTPBOOT folder /var/lib/tftp!!!
WARNING: Skip file copy to TFTPBOOT folder!!!

****** Xilinx Program Flash
****** Program Flash v2020.1 (64-bit)
  **** SW Build 2902540 on Wed May 27 19:54:35 MDT 2020
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.

WARNING: Failed to connect to hw_server at TCP:localhost:3121
Attempting to launch hw_server at TCP:localhost:3121

Connected to hw_server @ TCP:localhost:3121
Available targets and devices:
Target 0 : jsn-DLC10-00001ced722801
    Device 0: jsn-DLC10-00001ced722801-4ba00477-0

Retrieving Flash info...

Initialization done, programming the memory
Using default mini u-boot image file - /home/bss/Projects/DEVO/tools/Xilinx/Vitis/2020.1/data/xicom/cfgmem/uboot/zynq_qspi_x4_single.bin
===== mrd->addr=0xF800025C, data=0x00000000 =====
BOOT_MODE REG = 0x00000000
Downloading FSBL...
Running FSBL...
Finished running FSBL.
===== mrd->addr=0xF8000110, data=0x000FA220 =====
READ: ARM_PLL_CFG (0xF8000110) = 0x000FA220
===== mrd->addr=0xF8000100, data=0x00028008 =====
READ: ARM_PLL_CTRL (0xF8000100) = 0x00028008
===== mrd->addr=0xF8000120, data=0x1F000200 =====
READ: ARM_CLK_CTRL (0xF8000120) = 0x1F000200
===== mrd->addr=0xF8000118, data=0x001452C0 =====
READ: IO_PLL_CFG (0xF8000118) = 0x001452C0
===== mrd->addr=0xF8000108, data=0x0001E008 =====
READ: IO_PLL_CTRL (0xF8000108) = 0x0001E008
Info:  Remapping 256KB of on-chip-memory RAM memory to 0xFFFC0000.
===== mrd->addr=0xF8000008, data=0x00000000 =====
===== mwr->addr=0xF8000008, data=0x0000DF0D =====
MASKWRITE: addr=0xF8000008, mask=0x0000FFFF, newData=0x0000DF0D
===== mwr->addr=0xF8000910, data=0x000001FF =====
===== mrd->addr=0xF8000004, data=0x00000000 =====
===== mwr->addr=0xF8000004, data=0x0000767B =====
MASKWRITE: addr=0xF8000004, mask=0x0000FFFF, newData=0x0000767B

U-Boot 2020.01-08125-g1c9cef3 (May 05 2020 - 15:07:49 -0600)

Model: Zynq CSE QSPI SINGLE Board
DRAM:  256 KiB
WARNING: Caches not enabled
In:    dcc
Out:   dcc
Err:   dcc
Zynq> sf probe 0 0 0
Warning: SPI speed fallback to 100 kHz
SF: Detected is25lp128 with page size 256 Bytes, erase size 64 KiB, total 16 MiB
Zynq> Sector size = 65536.
f probe 0 0 0
Performing Erase Operation...
sf erase 0 FD0000
SF: 16580608 bytes @ 0x0 Erased: OK
Zynq> Erase Operation successful.
INFO: [Xicom 50-44] Elapsed time = 32 sec.
Performing Program Operation...
0%...sf write FFFC0000 0 20000
device 0 offset 0x0, size 0x20000
SF: 131072 bytes @ 0x0 Written: OK
Zynq> sf write FFFC0000 20000 20000
device 0 offset 0x20000, size 0x20000
SF: 131072 bytes @ 0x20000 Written: OK
Zynq> sf write FFFC0000 40000 20000
device 0 offset 0x40000, size 0x20000
SF: 131072 bytes @ 0x40000 Written: OK
Zynq> sf write FFFC0000 60000 20000
device 0 offset 0x60000, size 0x20000
SF: 131072 bytes @ 0x60000 Written: OK
[SNIP]
Zynq> sf write FFFC0000 F80000 20000
device 0 offset 0xf80000, size 0x20000
SF: 131072 bytes @ 0xf80000 Written: OK
Zynq> sf write FFFC0000 FA0000 20000
device 0 offset 0xfa0000, size 0x20000
SF: 131072 bytes @ 0xfa0000 Written: OK
Zynq> 100%
sf write FFFC0000 FC0000 12C
device 0 offset 0xfc0000, size 0x12c
SF: 300 bytes @ 0xfc0000 Written: OK
Zynq> Program Operation successful.
INFO: [Xicom 50-44] Elapsed time = 523 sec.

Flash Operation Successful

As I mentioned at the top of this post, the FSBL appears to halt after "Partition Header Offset:0x88888888". However, I can set the boot mode jumper on our custom board to JTAG and then boot Linux using the Xilinx Programming Cable and xsct.

bss@bss-VirtualBox:~/Projects/DEVO/DEVO_Petalinux/test/01_jtag_uboot$ cat boot_jtag_uboot.sh 
#!/bin/sh
SCRIPTDIR="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd -P)"
if ! command -v xsct > /dev/null ; then
    echo >&2 'Missing xsct, make sure environment is configured for xilinx.'
    exit 1
fi
xsct "${SCRIPTDIR}/boot_jtag_uboot.tcl"
bss@bss-VirtualBox:~/Projects/DEVO/DEVO_Petalinux/test/01_jtag_uboot$ cat boot_jtag_uboot.tcl 
#!/bin/env xsct
# You can run 'xsdb boot_jtag_INITRD_MINIMAL.tcl' to execute"

connect
targets -set -nocase -filter {name =~ "arm*#0"}

catch {stop}

set mctrlval [string trim [lindex [split [mrd 0xF8007080] :] 1]]
puts "mctrlval=$mctrlval"

puts stderr "INFO: Downloading ELF file: ./zynq_fsbl.elf to the target."
dow  "./zynq_fsbl.elf"
after 2000 ; con

# Give FSBL time to setup.
after 3000; stop

puts stderr "INFO: Downloading ELF file: ./u-boot.elf to the target."
dow  "./u-boot.elf"
after 2000; con

# Give uboot time to setup, but stop before it runs bootcmd.
after 1000 ; stop

# Let it continue to bootcmd.
con

exit

bss@bss-VirtualBox:~avnet_petalinux/test/01_jtag_uboot$ bash ./boot_jtag_uboot.sh 
attempting to launch hw_server

****** Xilinx hw_server v2020.1
  **** Build date : May 27 2020 at 20:33:44
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.

INFO: hw_server application started
INFO: Use Ctrl-C to exit hw_server application

INFO: To connect to this hw_server instance use url: TCP:127.0.0.1:3121

mctrlval=30800100
INFO: Downloading ELF file: ./zynq_fsbl.elf to the target.
INFO: Downloading ELF file: ./u-boot.elf to the target.

From the Linux boot log, I can see that the kernel recognizes this layout on the QSPI:

spi-nor spi0.0: found is25lp128, expected m25p80
spi-nor spi0.0: is25lp128 (16384 Kbytes)
4 fixed-partitions partitions found on MTD device spi0.0
Creating 4 MTD partitions on "spi0.0":
0x000000000000-0x000000ff0000 : "boot"
0x000000300000-0x000000fc0000 : "kernel"
0x000000fc0000-0x000001000000 : "bootenv"               
0x000001000000-0x000001000000 : "spare"
mtd: partition "spare" is out of reach -- disabled

From project-spec/configs/config

Notably, there is:
#
# Flash Settings
#
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_SELECT=y
# CONFIG_SUBSYSTEM_FLASH_MANUAL_SELECT is not set
# CONFIG_SUBSYSTEM_FLASH__ADVANCED_AUTOCONFIG is not set
#
# partition 0
#
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART0_NAME="boot"
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART0_SIZE=0x300000
#
# partition 1
#
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART1_NAME="kernel"
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART1_SIZE=0xCC0000
#
# partition 2
#
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART2_NAME="bootenv"
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART2_SIZE=0x40000
#
# partition 3
#
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART3_NAME="spare"
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART3_SIZE=0x0
#
# partition 4
#
CONFIG_SUBSYSTEM_FLASH_PS7_QSPI_0_BANKLESS_PART4_NAME=""
CONFIG_SUBSYSTEM_FLASH_IP_NAME="ps7_qspi_0"

FSBL reports:

Xilinx First Stage Boot Loader 
Release 2020.1  Apr 14 2021-07:23:49
Devcfg driver initialized 
Silicon Version 3.1
Boot mode is QSPI
Single Flash Information
FlashID=0x9D 0x60 0x18
ISSI 128M Bits
QSPI is in single flash connection
QSPI is in 4-bit mode
QSPI Init Done 
Flash Base Address: 0xFC000000
Reboot status register: 0x60400000
Multiboot Register: 0x0000C000
Image Start Address: 0x00000000
Partition Header Offset:0x88888888

I am concerned whether Partition Header Offset:0x88888888 looks correct?

Avnet / petalinux

FSBL fails during QSPI boot #33