A5voice attempt with Deluxe

Initial impressions are that the Deluxe has a longer length for A5 voice. Hexdump output was not the same as on the Explorer, and I hit an error when trying to dump the binary with A5voice.py. It's possible that I did a step wrong, so I'll outline them all here.

Platform details:

Windows 11, version 23H2, OS build 22631.3737
Python 3.11.6 with fresh virtual env (only has pip, setuptools, and construct installed)

Steps taken:

Downloaded Hydrasynth_Deluxe_Firmware_Pack_2.0.0.zip with sha256sum 00b8c2de519f33448c9987b639495fbecdad4f46de711df9220e5145c98a9a29.
Installed Hydrasynth Updater 2.1.0 (and drivers).
Copied libusb0.dll to tools directory (dev tools failed otherwise).
Opened tools/HydrasynthUpdaterForDevelopment.exe.
Boot-loaded Deluxe and connected to USB.
Ran upload with A5 voice option and length 0x01600000, saved to file. Note, there were numbers in the drop-down (1082060 and 1101200), but I didn't try these.
Hexdump gives the following for the last few lines:

15fffb0  0b fd 7c fd fb fd 72 fe c6 fe e8 fe dd fe ba fe  .²|²√²r■╞■Φ■▌■║■
15fffc0  97 fe 84 fe 7f fe 78 fe 58 fe 12 fe ae fd 3e fd  ù■ä■.■x■X■.■«²>²
15fffd0  de fc 9e fc 7f fc 6f fc 5a fc 33 fc fb fb c6 fb  ▐ⁿ₧ⁿ.ⁿoⁿZⁿ3ⁿ√√╞√
15fffe0  aa fb b4 fb e2 fb 22 fc 5b fc 7c fc 84 fc 85 fc  ¬√┤√Γ√"ⁿ[ⁿ|ⁿäⁿàⁿ
15ffff0  94 fc bf fc 04 fd 51 fd 8e fd ab fd a6 fd 8f fd  öⁿ┐ⁿ.²Q²Ä²½²ª²Å²

Cloned repo, added construct dependency.
python A5Voice.py --dump a5_voice_0x01600000.bin -d Parsing 'a5_voice_0x01600000.bin' gave this error:

Traceback (most recent call last):
  File "C:\Users\mikeb\Documents\repos\A5Voice\A5Voice.py", line 321, in <module>
    main()
  File "C:\Users\mikeb\Documents\repos\A5Voice\A5Voice.py", line 227, in main
    a5voice = A5VOICE.parse(data)
              ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 404, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 416, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 3099, in _parse
    raise CheckError("check failed during parsing", path=path)
construct.core.CheckError: Error in path (parsing) -> wavetable
check failed during parsing

Thank you for your testing, I think that you are correct that the Delux must use a larger segment for it's A5Voice.

The Struct() is parsed sequentically, so it looks like the first parts actually matched OK but then the Check() in 'Wavetable' failed.

GreedyRange() takes a template (for a better word) and then matches as many as it can, I suspect that the larger segment meant that it was trying to read data beyond the end of the file for some 'waves' and these failed. Thus the number of actually read 'waveforms' was lower than that declared in the header.

You can try retrieving a larger segment, the script will not care if it's oversized, but there are some changes need to be made....

Disasable the 'footer'.

The 'footer' points to a fixed location, which we know is wrong. Just comment it out. https://github.com/mungewell/A5Voice/blob/main/A5Voice.py#L140

A5VOICE = Struct(
        "header" / HEAD,
        "counting" / Padded(0x00000b08 - 0x00000020, COUNTING, pattern=b"\x88"),
        "wavetable" / WAVETABLE,
        #"footer" /  Pointer(0x015ffffc, FOOT)
)

Don't attempt to read 'samples'

The table structure precedes the actual waveform data, and we believe that we have all of the table. The 'wave' uses a 'Pointer()' to read it.

Comment out the pointer, the script will/should still parse the table OK. https://github.com/mungewell/A5Voice/blob/main/A5Voice.py#L57C1-L60C12

        #"sample" / Pointer(this.add1, Struct(
        #        "data" / Bytes(this._.size),
        #        "uns" / Int32ul,
        #)),

Allow Greedy flexibility

The Check() specifically wants a defined number of waveforms, and in other segment waves. Just comment them out, and script will accept quantity whatever it reads. https://github.com/mungewell/A5Voice/blob/main/A5Voice.py#L131

        #Check(len_(this.waveforms) == this._.header.waveform_count)

and https://github.com/mungewell/A5Voice/blob/main/A5Voice.py#L105

        #Check(len_(this.waves) == 17),

After making these changes you can see if the script works better, --dump should display information about the segment as text. It may be that you get another Check() error (this can be commented out freely), or maybe now it will be a Const() error were a specified string of data was not read. Please post any errors you get.

To comment further; once these bits are commented out, the last item of --dump will give you an idea of the size of the whole segment.

simon@the-void:~/A5Voice-github$ python3 A5Voice.py -d a5_voice_0x01600000.bin
...
                    Container: 
                        add1 = 22149184
                        add2 = 22149184
                        end = 22153280
                        size = 4096
                        wv0 = 249
                        wv1 = 255
                        wv2 = 3328

ie End = 22153280 = 0x01520840, round up to 0x01600000 (for Explorer).

Followed your prompts for commenting out stuff and the last line of the dump gave:

Container:
 add1 = 19649312
 add2 = 19649312
 end = 19665696
 size = 16384
 wv0 = 249
 wv1 = 255
 wv2 = 3328

That's 0x12C1320 in hex, I believe? Seems off.,

Firstly, this is excellent news as it means that the segment structure is somewhat close to the that on the Explorer.

I assume that you are still using the (known to be small) segment. That's OK all of the table is located before the samples... which start at least from '19649312'.

The actual start (add1_0_0) will be in the first few lines of the --dump

$ python3 A5Voice.py -d a5_voice_0x01600000.bin | head
Parsing 'a5_voice_0x01600000.bin'
Container: 
    header = Container: 
        add1_0_0 = 406752
        waveform_count = 220
    counting = Container: 
        count = ListContainer: 
            Container: 
                count1 = 137
            Container:

From the your above data snippet we can see that the size of that sample is 16K. For Explorer there are 17 'wave' samples per 'waveform', sized "[16k,16k,8k,4k,4k,4k,4k,4k,4k,4k,4k]", note that the last is 4K. So (put's on Sherlock hat) which 'wave' is it?

Is this the first (and perhaps only one) 'wave' of a 'waveform', or perhaps there is only one 'waveform' - meaning that the parsing hit another glitch/Check() issue.

The --dump result contains the structure of the data, with Containers for each level. I also included the 'index' parameter which counts the 'waveforms'. With a bit of grep, we have (for Explorer):

$ python3 A5Voice.py -d a5_voice_0x01600000.bin | grep -e "Container" -e "index"
...
                    Container: 
                    Container: 
                    Container: 
            Container: 
                index = 219
                waves = ListContainer: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container: 
                    Container:

Head

Container:
    header = Container:
        add1_0_0 = 409760
        waveform_count = 220
    counting = Container:
        count = ListContainer:
            Container:
                count1 = 137

And grep

                    Container:
                    Container:
                    Container:
            Container:
                index = 68
                waves = ListContainer:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:
                    Container:

So the table says it has the 220 'waveforms' and it looks like 17 'waves' each with 16KByte samples.

End of file would be approx: 409760 + (220 ( 17 (16384 + 32))) = 61805600 = 0x3AF1420 = (round up to) 0x3b00000 or 0x4000000 ???

With the commented out section (ie no samples stored/processed), if I limit the Explorer to just 69 'waveforms' then the table is complete at address 0x0001f9e4. Which is way 'earlier' than the end of your file... as expected.

$ python3 A5Voice.py -o test.bin -W 69 -D a5_voice_0x01600000.bin

0001f9a0  f9 7f ff 7f 81 7f ff 80  3c 00 80 00 00 7f 00 00  |........<.......|
0001f9b0  3c 40 2c 00 00 0d 00 00  20 00 00 00 ff 7f 00 00  |<@,..... .......|
0001f9c0  3c 00 00 00 00 00 7f 00  1e 00 00 7f 00 1f 3c 00  |<.............<.|
0001f9d0  7f 80 7f 80 7f 7f 7f 7f  00 7f 00 00 20 1e 00 00  |............ ...|
0001f9e0  00 7f 7f 7f                                       |....|
0001f9e4

This likely means that some table content/vaules don't match the template, and thus GreedyRange() only gives the 69 'waveforms'... something in waveform 70 is wonky.

We can force parsing to continue by replacing GreedyRange() with Array() as follows, this WILL cause a parse error but now it will tell you why.

https://github.com/mungewell/A5Voice/blob/main/A5Voice.py#L103


        "waves" / Array(17, WAVE),                    # or should we use 'Array(17, WAVE)'?

        #Check(len_(this.waves) == 17),

and https://github.com/mungewell/A5Voice/blob/main/A5Voice.py#L129C1-L131C68

        "waveforms" / Array(220, WAVEFORM),            # or should we use 'this._.waveform_count'?

        #Check(len_(this.waveforms) == this._.header.waveform_count)

I expect that this will give an error on one of the Const() statements, and that we'll have to alter these so that script will allow/remember variable values.

$ C:\Users\mikeb\Documents\repos\A5Voice> python .\A5Voice.py -d .\a5_voice_0x01600000.bin
Parsing '.\a5_voice_0x01600000.bin'
Traceback (most recent call last):
  File "C:\Users\mikeb\Documents\repos\A5Voice\A5Voice.py", line 321, in <module>
    main()
  File "C:\Users\mikeb\Documents\repos\A5Voice\A5Voice.py", line 227, in main
    a5voice = A5VOICE.parse(data)
              ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 404, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 416, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2530, in _parse
    e = self.subcon._parsereport(stream, context, path)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mikeb\Documents\repos\A5Voice\venv\Lib\site-packages\construct\core.py", line 2845, in _parse
    raise ConstError(f"parsing expected {repr(self.value)} but parsed {repr(obj)}", path=path)
construct.core.ConstError: Error in path (parsing) -> wavetable -> waveforms
parsing expected b'\x00\x00\x08\x7f' but parsed b'\x00\x00\x00\x00'

As expected the Const() threw an error, but I'm not sure what this actually means wrt the table and how best to figure out. As above I can limit my table to just 69 waveforms, and that appears finish at 0x0001f9e4.

If I search for the data expected for waveform 70 (ie the one that's erroring), i get...

            Container: 
                index = 69
                un0 = 32411
                un1 = 32436
                un2 = 32461

32411 = 0x7E9B, reverse to search as we're little endian...

$ python3 ~/SearchBin-github/searchbin.py -p "00 00 08 7f 9B 7E" a5_voice_0x01600000.bin 
Match at offset:         129508        1F9E4 in  a5_voice_0x01600000.bin

$ hexdump -C a5_voice_0x01600000.bin | grep -B 5 -A 5 "0001f9e0"
0001f990  60 47 6e 00 00 00 ff 7f  80 64 00 00 00 64 00 00  |`Gn......d...d..|
0001f9a0  f9 7f ff 7f 81 7f ff 80  3c 00 80 00 00 7f 00 00  |........<.......|
0001f9b0  3c 40 2c 00 00 0d 00 00  20 00 00 00 ff 7f 00 00  |<@,..... .......|
0001f9c0  3c 00 00 00 00 00 7f 00  1e 00 00 7f 00 1f 3c 00  |<.............<.|
0001f9d0  7f 80 7f 80 7f 7f 7f 7f  00 7f 00 00 20 1e 00 00  |............ ...|
0001f9e0  00 7f 7f 7f 00 00 08 7f  9b 7e 00 00 08 00 10 7f  |.........~......|  <---------
0001f9f0  b4 7e 00 00 10 00 18 7f  cd 7e 00 00 18 00 20 7f  |.~.......~.... .|
0001fa00  e6 7e 00 00 20 00 28 7f  ff 7e 00 00 28 00 30 7f  |.~.. .(..~..(.0.|
0001fa10  18 7f 00 00 30 00 37 7f  31 7f 00 00 37 00 3e 7f  |....0.7.1...7.>.|
0001fa20  4a 7f 00 00 3e 00 47 7f  63 7f 00 00 47 00 4f 7f  |J...>.G.c...G.O.|
0001fa30  7c 7f 00 00 4f 00 57 7f  95 7f 00 00 57 00 5e 7f  ||...O.W.....W.^.|

I suggest that you have a look at the same/similar location to see if anything looks drastically different.

I appreciate that progress has slowed, but at least we have confirmed that the basic struct is the same as the Explorer. It appears that some tweaks are needed, and that the segment as a whole is larger. Did you attempt to download a larger segment?

mungewell / A5Voice