[rom_ext] Prodc rom_ext binaries for cw310 do not work

[jettr]

Description

Using the latest ROM_EXT binaries on the earlgrey_es_sival branch at sw/device/silicon_creator/rom_ext/prodc/binaries/rom_ext_real_prod_signed_slot_a_fpga_cw310.signed.bin, I get the following boot error

ROM:2ac41496
BFV:01525202
LCV:21084210
VER:4f70656e

I didn't change the bitstream of ROM at all. I also tried the binaries in the sival folder instead of prodc, and then I got BFV:01535603 error instead.

This is probably user error, but I don't know what I am doing wrong

[jesultra]

For FPGA, you should use another ROM_EXT, one that does not have silicon_creator in its name, but instead fpga_cw310.

[jettr]

Sorry copy and paste bug in the description, I updated the description. I actually meant this file sw/device/silicon_creator/rom_ext/prodc/binaries/rom_ext_real_prod_signed_slot_a_fpga_cw310.signed.bin

[moidx]

Hi @jettr, this seems to be the error:

https://github.com/lowRISC/opentitan/blob/d3942ca0743eee5616c63d5efd38e91fb4949f79/sw/device/silicon_creator/lib/drivers/retention_sram.c#L56-L61

Probably related to this change https://github.com/lowRISC/opentitan/pull/21587

Need to check what is causing the mismatch with ROM. Is the bitstream being loaded from the OpenTitan bitstream cache?

[jettr]

It is the same bitstream that works with the previous ROM_EXT for cw310. Unfortunately it looks like it was a hand build bitstream: www.crrev.com/i/6944790 (sorry for internal link).

I don't mind using a different bitstream that emulate ES silicon. Can you point me to one?

[moidx]

I usually get the bitstream closest to my latest commit by using the --define bitstream=gcp_splice flag. For example:

$ bazel test --test_output=streamed \
  --define bitstream=gcp_splice \
  --cache_test_results=no  \
  --local_test_jobs 1 \
  //sw/device/silicon_creator/rom/e2e:rom_e2e_smoke_fpga_cw310_sival_rom_ext

Invoking test: sw/host/opentitantool/opentitantool --rcfile= --logging=info --interface=hyper310 --exec=transport init --exec=fpga load-bitstream hw/bitstream/universal/splice.bit --exec=bootstrap --clear-uart=true sw/device/silicon_creator/rom/e2e/rom_e2e_smoke_fpga_cw310_sival_rom_ext.img --exec=console --non-interactive --exit-success='PASS.*\n' --exit-failure='((FAIL|FAULT).*\n)|(BFV:[0-9a-f]{8})' no-op

The fetched bitstream is hw/bitstream/universal/splice.bit.

$ find bazel-out/ -name "splice.bit"
bazel-out/k8-fastbuild-ST-0ead4726db73/bin/hw/bitstream/universal/splice.bit
bazel-out/k8-fastbuild-ST-2cc462681f62/bin/sw/device/silicon_creator/rom/e2e/rom_e2e_smoke_fpga_cw310_sival_rom_ext.bash.runfiles/lowrisc_opentitan/hw/bitstream/universal/splice.bit
bazel-out/k8-fastbuild-ST-2cc462681f62/bin/sw/device/silicon_creator/rom/e2e/rom_e2e_smoke_fpga_cw310_sival.bash.runfiles/lowrisc_opentitan/hw/bitstream/universal/splice.bit

If this doesn't work because of lack of Vivado install in your environment, you may be able to get the bitstream from one of the most recent post-submit CI runs.

@cfrantz do you know of the top of your hand if we have any rules to fetch bitstreams from the cache from the command line?

[cfrantz]

We don't have any rules to just fetch bitstreams and none of the cached bitstreams are going have their OTP setup for the prodc customer.

You can create such a bitstream with a command like this:

bazel build //hw/bitstream/universal:splice \
       --//hw/bitstream/universal:env=//hw/top_earlgrey:fpga_cw310_sival \
       --//hw/bitstream/universal:rom=//sw/device/silicon_creator/rom:rom_with_real_keys \
       --//hw/bitstream/universal:otp=//hw/ip/otp_ctrl/data/earlgrey_a0_skus/prodc:otp_img_prod_manuf_personalized

That should allow a pre-signed ROM_EXT buildt for cw310 to run (the files are committed here: sw/device/silicon_creator/rom_ext/prodc/binaries/....

I think I gave similar instructions on how to run prodc code on the FPGA to @jesultra, so he can probably help out here (because, like an idiot, I forgot to write the instructions down for myself).

[a-will]

This doesn't matter so much for the earlgrey_es_sival branch, but if you've been using a given bazel workspace across commits on a branch with active hardware development, note that you'll likely need to sync the @bitstream repo manually when HEAD changes:

bazel sync --configure

The bitstreams_repo repository_rule does not depend on much of anything other than the two environment variables BAZEL_BITSTREAMS_CACHE and BITSTREAM, in addition to the bitstreams_workspace.py script.

I think I gave similar instructions on how to run prodc code on the FPGA to @jesultra, so he can probably help out here (because, like an idiot, I forgot to write the instructions down for myself).

You wrote a similar set of instructions down for everyone, so it's close! It's just the fake-signed variant that is documented in the BUILD file, though: https://github.com/lowRISC/opentitan/blob/11c4976ed0eb7f442f0dc7c9bdfb448ec099bb2c/sw/device/silicon_creator/rom_ext/prodc/BUILD#L28-L34

[jettr]

I have been limping along without installing Vivado, but I finally installed it this morning. Hopefully I have everything to build, but I am still running into failures

Here is what I run:

$ bazel sync --configure
WARNING:root:Closest bitstream to HEAD is 89204128efb5ff9e1c29570b999b27c3a90756f5.
 checking cached actions

$ bazel build //hw/bitstream/universal:splice
DEBUG: <path_to_opentitan>/rules/opentitan/splice.bzl:101:14: @//hw/bitstream/universal:splice: No exec_env.  Nothing to do.
INFO: Analyzed target //hw/bitstream/universal:splice (3 packages loaded, 1024 targets configured).
INFO: Found 1 target...
INFO: Elapsed time: 1.058s, Critical Path: 0.06s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

$ bazel build //hw/bitstream/universal:splice \
       --//hw/bitstream/universal:env=//hw/top_earlgrey:fpga_cw310_sival \
       --//hw/bitstream/universal:rom=//sw/device/silicon_creator/rom:rom_with_real_keys \
       --//hw/bitstream/universal:otp=//hw/ip/otp_ctrl/data/earlgrey_a0_skus/prodc:otp_img_prod_manuf_personalized
INFO: Build options --//hw/bitstream/universal:env, --//hw/bitstream/universal:otp, and --//hw/bitstream/universal:rom have changed, discarding analysis cache.
DEBUG: <path_to_opentitan>/rules/autogen.bzl:86:14: NOTE: stamping is disabled, the chip_info section will use a fixed version string
INFO: Analyzed target //hw/bitstream/universal:splice (38 packages loaded, 31513 targets configured).
INFO: Found 1 target...
INFO: From GenVivadoImage hw/bitstream/universal/splice-rom.update.mem:
2024-05-03 09:02:25,077 [gen_vivado_mem_image.py] Generating updatemem-compatible MEM file for ROM.
ERROR: <path_to_opentitan>/hw/bitstream/universal/BUILD:29:17: SpliceBitstream hw/bitstream/universal/splice-rom.bit failed: (Exit 1): updatemem failed: error executing command (from target //hw/bitstream/universal:splice) updatemem --force --meminfo external/bitstreams/cache/89204128efb5ff9e1c29570b999b27c3a90756f5/chip_earlgrey_cw310_hyperdebug/rom.mmi --data ... (remaining 7 arguments skipped)
src/main/tools/process-wrapper-legacy.cc:80: "execvp(updatemem, ...)": No such file or directory
INFO: Elapsed time: 2.079s, Critical Path: 1.14s
INFO: 9 processes: 8 internal, 1 local.
FAILED: Build did NOT complete successfully

I don't know what I am missing at this point.

[a-will]

Notice execvp() fails because it can't find the updatemem binary. It's not on your PATH.

Make sure you source the settings64.sh script in the installation directory to set the required environment variables.

[jettr]

That helped, but now I get different errors:

$ bazel build //hw/bitstream/universal:splice \
       --//hw/bitstream/universal:env=//hw/top_earlgrey:fpga_cw310_sival \
       --//hw/bitstream/universal:rom=//sw/device/silicon_creator/rom:rom_with_real_keys \
       --//hw/bitstream/universal:otp=//hw/ip/otp_ctrl/data/earlgrey_a0_skus/prodc:otp_img_prod_manuf_personalized
INFO: Build options --//hw/bitstream/universal:otp and --//hw/bitstream/universal:rom have changed, discarding analysis cache.
DEBUG: <path_to_opentitan>/rules/autogen.bzl:86:14: NOTE: stamping is disabled, the chip_info section will use a fixed version string
INFO: Analyzed target //hw/bitstream/universal:splice (0 packages loaded, 31513 targets configured).
INFO: Found 1 target...
INFO: From GenVivadoImage hw/bitstream/universal/splice-rom.update.mem:
2024-05-03 09:34:06,303 [gen_vivado_mem_image.py] Generating updatemem-compatible MEM file for ROM.
INFO: From SpliceBitstream hw/bitstream/universal/splice-rom.bit:
ERROR: [Updatemem 57-44] xc7k410t device is not supported by update_mem.

****** updatemem v2021.1 (64-bit)
  **** SW Build 3247384 on Thu Jun 10 19:36:07 MDT 2021
  **** IP Build 3246043 on Fri Jun 11 00:30:35 MDT 2021
    ** Copyright 1986-2021 Xilinx, Inc. All Rights Reserved.

source /opt/tools/Xilinx/Vivado/2021.1/scripts/updatemem/main.tcl -notrace
Command: update_mem -meminfo external/bitstreams/cache/89204128efb5ff9e1c29570b999b27c3a90756f5/chip_earlgrey_cw310_hyperdebug/rom.mmi -data bazel-out/k8-fastbuild/bin/hw/bitstream/universal/splice-rom.update.mem -proc dummy -bit external/bitstreams/cache/89204128efb5ff9e1c29570b999b27c3a90756f5/chip_earlgrey_cw310_hyperdebug/lowrisc_systems_chip_earlgrey_cw310_hyperdebug_0.1.bit -out bazel-out/k8-fastbuild/bin/hw/bitstream/universal/splice-rom.bit -force
Loading bitfile external/bitstreams/cache/89204128efb5ff9e1c29570b999b27c3a90756f5/chip_earlgrey_cw310_hyperdebug/lowrisc_systems_chip_earlgrey_cw310_hyperdebug_0.1.bit
0 Infos, 0 Warnings, 0 Critical Warnings and 1 Errors encountered.
update_mem failed
update_mem: Time (s): cpu = 00:00:17 ; elapsed = 00:00:16 . Memory (MB): peak = 2020.332 ; gain = 757.418 ; free physical = 35421 ; free virtual = 209214
ERROR: [Common 17-39] 'update_mem' failed due to earlier errors.

INFO: [Common 17-206] Exiting updatemem at Fri May  3 09:34:23 2024...
ERROR: <path_to_opentitan>/hw/bitstream/universal/BUILD:29:17: output 'hw/bitstream/universal/splice-rom.bit' was not created
ERROR: <path_to_opentitan>/hw/bitstream/universal/BUILD:29:17: SpliceBitstream hw/bitstream/universal/splice-rom.bit failed: not all outputs were created or valid
INFO: Elapsed time: 18.916s, Critical Path: 18.04s
INFO: 4 processes: 2 internal, 2 local.
FAILED: Build did NOT complete successfully

Looks like I don't have the right support installed? I followed the guide to install Vivado 2021.1 with Kintex-7 support. Maybe that is out of date?

[a-will]

Hm... that's peculiar. I haven't used that particular version before... I wonder if there is an incorrectly-placed license check or something.

Also, FYI, if you are not developing FPGA designs / building the base bitstreams, you would only need the "Lab Edition." I just installed and tried that one, and it worked fine on the bitstream you used.

[jettr]

Thank you all for the help on this so far! I still can't build it, but there are better things we can be spending our time on. I will just get someone to build binaries for me when needed. Closing issue.