xtensa FLIX support

[shuffle2]

Hi, I recently found https://github.com/flto/ghidra-xtensa-flix after getting a firmware targeting "HiFi4" DSP (https://www.cadence.com/en_US/home/resources/product-briefs/tensilica-hifi-dsp-family-pb.html) and noticing that ghidra does not handle the FLIX-encoded instructions.

To summarize, one of xtensa's main features is that it allows adding custom instructions. FLIX builds on this by providing a standard way to encode VLIW instructions. Some FLIX-encoded instructions are "standard" in the sense that they're predefined and provided along with a given xtensa core, depending on the feature set. A licensee can still replace any FLIX instruction with their own design, but normally this doesn't happen. Generally, FLIX instruction is sized by the number of slots it has (which can execute in parallel). The actual instructions are encoded into the specific slots, with certain slots having limitations.

After trying to port the above repo to current ghidra, I was disappointed to find it didn't Just Work. I think this is because the existing code gives flixinsn a size of 64 bits, whereas my firmware happens to be using 6 and 11 byte instructions. For the record, I'm using xt-objdump.exe --xtensa-core=hifi4_ss_spfpu_7 -m xtensa ..., and it gives an output something like:

5ff025cc <.data+0x25cc>:
5ff025cc:       3601b2                          entry   a1, 0x5900
5ff025cf:       72d101                          addmi   a7, a1, 0x100
5ff025d2:       72c7d0                          addi    a7, a7, -48
5ff025d5:       e2d724                          addmi   a14, a7, 0x2400
5ff025d8:       fec7471853cc                    { addmi a15, a7, 0x1c00; addmi  a8, a7, 0x1400 }
5ff025de:       ae47c70d93cc                    { addmi a10, a7, 0x400; addmi   a13, a7, 0x2c00 }
5ff025e4:       9ec7470cd3cc                    { addmi a9, a7, 0xc00; addmi    a12, a7, 0x3400 }
5ff025ea:       be070c4c53c8                    { addmi a11, a7, 0x4000; addi   a12, a12, 16 }
5ff025f0:       ce610b1b59c8                    { s32i  a12, a1, 88; addi       a11, a11, 16 }
5ff025f6:       bed1090959c8                    { s32i  a11, a1, 52; addi       a9, a9, 16 }
5ff025fc:       9e010d1d59c8                    { s32i  a9, a1, 64; addi        a13, a13, 16 }
5ff02602:       de410a1a59c8                    { s32i  a13, a1, 80; addi       a10, a10, 16 }
5ff02608:       aef1080859c8                    { s32i  a10, a1, 60; addi       a8, a8, 16 }
5ff0260e:       8e110f1f59c8                    { s32i  a8, a1, 68; addi        a15, a15, 16 }
5ff02614:       fe210e1e59c8                    { s32i  a15, a1, 72; addi       a14, a14, 16 }
5ff0261a:       ee81612f99c8                    { s32i  a14, a1, 160; addi      a15, a1, 38 }
5ff02620:       f26113                          s32i    a15, a1, 76
...
5ff026fe:       256803                          call8   0x5ff05d80
5ff02701:       a6f340                          blti    a3, 0x100, 0x5ff02745
5ff02704:       a6f43d                          blti    a4, 0x100, 0x5ff02745
5ff02707:       a6f53a                          blti    a5, 0x100, 0x5ff02745
5ff0270a:       fe76e13900a7                    { blti.w15      a6, 0x100, 0x5ff02745; l32i.n   a9, a1, 56 }
5ff02710:       0fc9b10bda03d0000a8b00  { blti.w15      a9, 0x100, 0x5ff02745; l32i     a11, a1, 108; nop }
5ff0271b:       4fcba10cca2390000a8b00  { blti.w15      a11, 0x100, 0x5ff02745; l32i    a12, a1, 104; nop }
5ff02726:       8fcc910dda2350000a8b00  { blti.w15      a12, 0x100, 0x5ff02745; l32i    a13, a1, 100; nop }
5ff02731:       0fcd810eca0350000a8b00  { blti.w15      a13, 0x100, 0x5ff02745; l32i    a14, a1, 96; nop }
5ff0273c:       a62e05                          blti    a14, 2, 0x5ff02745
5ff0273f:       2eea901001b5                    { bgei.w15      a10, 2, 0x5ff02861; nop }
5ff02745:       2166f8                          l32r    a2, 5ff008e0 (0x5fe07e98)
5ff02748:       ae03000b1be0                    { movi  a10, 3; movi    a11, 0 }
5ff0274e:       f02000                          nop

So..to make this a question, do you have recommendations about how to go about "properly" adding support for FLIX? Is it worthwhile fixing up the linked plugin, or should I just start over?

note: original extension author points out FLIX encodings can be found in binutils: https://github.com/flto/ghidra-xtensa-flix/issues/1#issuecomment-1833784508

[GhidorahRex]

I can try to take a look at it. It looks like the default implementation of FLIX is 32/64-bit (I've found some documentation that assumes that), but they extended it to anything up to 128-bits. It's possible to extend it - you'd want to look at the format_decoder text in the binutils code you linked. I haven't delved deep enough into xtensa to understand what the different formats mean, but that's where I would start.