Need a simple "un-rimage" tool for build reproducibility

[marc-hb]

rimage signatures include the date and on some platforms also rely on a random salt value. SOF builds are already reproducible[]; this signature difference is currently the only difference that stops someone from quickly verifying that a random `sof-.ri` file has been built from the source version / git tag and toolchain it claims it has.

A simple unrimage tool will provide the most convenient and most scriptable solution to that problem: simply strip signatures from two sof.ri files and compare them, they should be 100% identical.

The best starting points seem to be either sof/tools//sof_ri_info/sof_ri_info.py or rimage itself (or both?)

PS: DEBUG builds need more work to be reproducible , see https://github.com/thesofproject/sof/pull/3979

[*]https://reproducible-builds.org/

[marc-hb]

Here's a very well tested rimage HACK to make sof.ri images reproducible locally. It's convenient to make sure some changes don't affect compiler output.

diff --git a/src/css.c b/src/css.c
index c86148b89015..b1a9f48cf216 100644
--- a/src/css.c
+++ b/src/css.c
@@ -23,7 +23,7 @@ void ri_css_v2_5_hdr_create(struct image *image)

        /* get local time and date */
        gettimeofday(&tv, NULL);
-       seconds = tv.tv_sec;
+       seconds = 1 ; // tv.tv_sec;
        date = localtime(&seconds);

        if (!date) {
@@ -80,7 +80,7 @@ void ri_css_v1_8_hdr_create(struct image *image)

        /* get local time and date */
        gettimeofday(&tv, NULL);
-       seconds = tv.tv_sec;
+       seconds = 1; //tv.tv_sec;
        date = localtime(&seconds);

        if (!date) {
@@ -137,7 +137,7 @@ void ri_css_v1_5_hdr_create(struct image *image)

        /* get local time and date */
        gettimeofday(&tv, NULL);
-       seconds = tv.tv_sec;
+       seconds = 1 ; // tv.tv_sec;
        date = localtime(&seconds);

        if (!date) {
diff --git a/src/pkcs1_5.c b/src/pkcs1_5.c
index 0bfb901f4cbb..ef381aa9ddca 100644
--- a/src/pkcs1_5.c
+++ b/src/pkcs1_5.c
@@ -313,7 +313,7 @@ int pkcs_v1_5_sign_man_v2_5(struct image *image,

        /* sign the manifest */
        ret = RSA_padding_add_PKCS1_PSS(priv_rsa, sig,
-                       digest, image->md, /* salt length */ 32);
+                       digest, image->md, /* salt length */ 0); // Do NOT do this 
        if (ret <= 0) {
                ERR_error_string(ERR_get_error(), path);
                fprintf(stderr, "error: failed to sign manifest %s\n", path);

[marc-hb]

From @RanderWang emails:

To get FW content, Read FW, the first bytes of FW is a structure of sof_ext_man_header. Then you can get the length of ext_man. Please check snd_sof_ext_man_size in kernel driver. Original FW text, data, bss is started at fw->start_addr + length of ext_man + 0x8000. Just check at this addr to the end.

And if there is no ext_man in fw, the you can check fw->start_addr + 0x8000 directly. Please check snd_sof_ext_man_size in kernel driver to judge whether there is ext_man or not.

we build sof-tgl & bootloader-tgl which are binary without any signature, then we will merge these two binaries into one binary with all signature.

[marc-hb]

Sample sof/tools/sof_ri_info/sof_ri_info.py output:

SOF Binary /lib/firmware/intel/sof/sof-apl.ri size 0x48000

  CSE Manifest ver 0x101 checksum 0x97 partition name ADSP

    ADSP.man (CSS Manifest) type 0x4 ver 0x10000 date 2020/09/28
      Rsvd0 0x0
      Modulus size (dwords) 64
        1f f4 58 74 64 d4 ae 90 ... b5 c2 49 4e 2a 5f 47 c2 (APL Intel prod key)
      Exponent size (dwords) 1
        01 00 01 00
      Signature
        26 a5 a8 e2 72 2e fd a4 ... 88 de 51 1f 58 5d 56 04

      Plat Fw Auth Extension name ADSP vcn 0x0 bitmap 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 svn 0x0

      Other Extension type 0x50534441 length 0x48000

    cavs0015.met (ADSP Metadata File Extension) ver 0x0 base offset 0x2000 limit offset 0x49bc0
      IMR type 0x4
      Attributes
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

    cavs0015

  cavs0015 (ADSP Manifest) name ADSPFW build ver 1.6.0.1 feature mask 0xffff image flags 0x0
    HW buffers base address 0x0 length 0x0
    Load offset 0x2000

    BRNGUP    0d7b48cc-1ea9-470a-a8c1-533424528a17
      entry point 0xb000a000 type 0x21 ( loadable LL )
      cfg offset 0 count 0 affinity 0x3 instance max count 1 stack size 0x1
      .text   0xb000a000 file offset 0x8000 flags 0x1001f ( contents alloc load readonly code type=0 pages=1 )
      .rodata 0xb0002000 file offset 0x9000 flags 0x1012f ( contents alloc load readonly data type=1 pages=1 )
      .bss    0x0 file offset 0x0 flags 0xf00 ( type=15 pages=0 )

    BASEFW    fc869e2e-45f8-4045-a416-89880ae320a9
      entry point 0xbe008400 type 0x21 ( loadable LL )
      cfg offset 0 count 0 affinity 0x3 instance max count 1 stack size 0x1
      .text   0xbe008000 file offset 0xa000 flags 0x2c001f ( contents alloc load readonly code type=0 pages=44 )
      .rodata 0xbe034000 file offset 0x36000 flags 0x12012f ( contents alloc load readonly data type=1 pages=18 )
      .bss    0xbe046000 file offset 0x0 flags 0x310202 ( alloc type=2 pages=49 )

Intel Apollolake
  imr                                 0xa0000000 (4194304)
  l2 hpsram                           0xbe000000 (454656 + 69632  86.72% used)
    BASEFW.text                         0xbe008000 (180224)
    BASEFW.rodata                       0xbe034000 (73728)
    BASEFW.bss                          0xbe046000 (200704)
  l2 lpsram                           0xbe800000 (131072)

[RanderWang]

@marc-hb I can implement it in rimage today. I am not familiar with python. what's your idea ? And the input is fw1, fw2 ? how about output ? a error message is enough for CI ?

[marc-hb]

@marc-hb I can implement it in rimage today. I am not familiar with python.

Having the feature in rimage sounds great, Thank you so much! Python is faster and safer to write... when you know it but I'm guessing sof_ri_info.py duplicates a lot of the rimage knowledge and copy/paste/diverge is not great either. Whatever works?

And the input is fw1, fw2 ? how about output ? a error message is enough for CI ?

I would really prefer to keep the "unrimage" feature super simple:

• input: some-sof.ri file
• outputs: original file(s) that were signed + (optional) signature

Please do not implement any comparison in C in rimage itself, there are much better tools already available. For CI cmp or diff are simple enough.

Diffoscope is pretty amazing, random example:

make -C installer/
PATH=~/SOF/xtensa-byt-elf/xtensa-byt-elf/bin/:$PATH
apt install diffoscope # many dependencies, takes a while
diffoscope --text-color=always installer-builds/build_byt_gcc/sof  installer-builds/build_cht_gcc/sof | less -R

Output sample:

│      Address    Length
-    ff2d3844 000002d6 
+    ff2d2834 00000068 
│      00000000 00000000 
│    Length:                   28
│    Version:                  2
│    Offset into .debug_info:  0x19f9
│    Pointer Size:             4
│    Segment Size:             0
│  
│      Address    Length
-    ff2d3f5c 00000049 
+    ff2d289c 00000049 
│      00000000 00000000
├── objdump --line-numbers --disassemble --demangle --reloc --section=.ResetVector.text {}
│ @@ -16,18 +16,18 @@
│  ff2c0018:    008000                  any4    b0, b0:b1:b2:b3
│  ff2c001b:    222200                  orb     b2, b2, b0
│  ff2c001e:    002522                  l32i    a2, a5, 0
│  ff2c0021:    e00000                  subx4   a0, a0, a0
│  ff2c0024:    2c00f4                  excw
│  ff2c0027:    00000000000000ff        { ae_sp16x2f.c  aep0, a0, a15; excw }
│  ff2c002f:    000000                  ill
-ff2c0032:    b80000                  excw
-ff2c0035:    2d2d                    excw
-ff2c0037:    000000ff2d2f9cff        { excw; excw }
-ff2c003f:    000000                  ill
+ff2c0032:    f80000                  excw
+ff2c0035:    ff2d16                  beqz    a13, ff2c002b <_ResetVector+0x2b>
+ff2c0038:    18dc                    bnez.n  a8, ff2c004d <_ResetHandler+0x9>
+ff2c003a:    ff2d                    break.n 15
│       ...
│  
│  ff2c0044 <_ResetHandler>:
│  _ResetHandler():
│  ff2c0044:    000c                    movi.n  a0, 0
│  ff2c0046:    13e400                  wsr.intenable   a0
│  ff2c0049:    ffee21                  l32r    a2, ff2c0004 <_ResetVector+0x4>
│ @@ -128,8 +128,8 @@
│  ff2c0152:    13d240                  wsr.excsave2    a4
│  ff2c0155:    ffb641                  l32r    a4, ff2c0030 <_ResetVector+0x30>
│  ff2c0158:    13d340                  wsr.excsave3    a4
│  ff2c015b:    ffb641                  l32r    a4, ff2c0034 <_ResetVector+0x34>
│  ff2c015e:    13d440                  wsr.excsave4    a4
│  ff2c0161:    ffb541                  l32r    a4, ff2c0038 <_ResetVector+0x38>
│  ff2c0164:    13d540                  wsr.excsave5    a4
-ff2c0167:    00a8c5                  call0   ff2c0bf4 <_start>
+ff2c0167:    00b205                  call0   ff2c0c88 <_start>
├── objdump --line-numbers --disassemble --demangle --reloc --section=.UserExceptionVector.literal {}
│ @@ -1,9 +1,8 @@
│  
│  
│  
│  Disassembly of section .UserExceptionVector.literal:
│  
│  ff2c0658 <.UserExceptionVector.literal>:
-ff2c0658:    16ec                    bnez.n  a6, ff2c067d <_DoubleExceptionVector_text_start+0x1>
-ff2c065a:    30                              .byte 0x30
+ff2c0658:    301874                  excw
│  ff2c065b:    ff                              .byte 0xff
├── objdump --line-numbers --disassemble --demangle --reloc --section=.text {}
│ @@ -12,15 +12,15 @@
│  ff2c06c4 <clock_platform_set_ssp_freq>:
│  clock_platform_set_ssp_freq():
│  ff2c06c4:    004136                  entry   a1, 32
│  ff2c06c7:    fffe81                  l32r    a8, ff2c06c0 <_stext>
│  ff2c06ca:    1133e0                  slli    a3, a3, 2
│  ff2c06cd:    883a                    add.n   a8, a8, a3
│  ff2c06cf:    08a8                    l32i.n  a10, a8, 0
-ff2c06d1:    0cfea5                  call8   ff2cd6bc <ipc_pmc_send_msg>
+ff2c06d1:    0bda65                  call8   ff2cc478 <ipc_pmc_send_msg>
│  ff2c06d4:    0a2d                    mov.n   a2, a10
│  ff2c06d6:    f01d                    retw.n
│  ff2c06d8:    340080                  extui   a0, a8, 0, 4

[RanderWang]

thanks! Great share.

[RanderWang]

@marc-hb please check https://github.com/thesofproject/rimage/pull/43. rimage -p sof-tgl.ri you will get no_sig_sof-tgl.ri without any signature

[marc-hb]

reproducibility test submitted in https://github.com/thesofproject/sof/pull/4829

[marc-hb]

--erase-vars feature implemented in https://github.com/thesofproject/sof/commit/f8ca8536fd772

reproducible.ri tests implemented and running in CI. Only for XTOS though, not for Zephyr yet.