search

mungewell / zoom-zt2

Python script to install/remove effects from the Zoom G1Four pedal
MIT License
50 stars 10 forks source link

What format are V6 effects? #37

Open mungewell opened 2 years ago

mungewell commented 2 years ago

https://zoomcorp.com/en/us/vocal-processors/vocal-processors/V6/v6-support/

136_ZDLF.txt

Seems that they are somewhat like a ZD2, but decode_effect complains.

mungewell commented 2 years ago

Removing the PRMJ/PRME block from the parser definition gets most (but not all) decoding... seems that they are all Group 29 (Vocal).

$ find . -name "*.zd2" -exec bash -c "python3 ~/zoom-zt2-sdw-github/decode_effect.py -s {} " \;
0x1dc00180 : VoVocoder (v0.01), ./1024006.zd2
0x1dc0ffff : CMN FFT (v0.01), ./106502.zd2
0x1dc00021 : PitchDtct (v0.01), ./1110022.zd2
0x1dc00011 : PSOLAInst (v0.01), ./1134598.zd2
0x1dc001d1 : Harmony (v0.01), ./1146886.zd2
0x1dc0fffe : CMN KALIB (v0.01), ./114694.zd2
0x1dc00029 : VunVDtct (v0.01), ./1159174.zd2
0x1dc0ffff : CMN FFT (v0.01), ./1179654.zd2
0x1dc0fffe : CMN KALIB (v0.01), ./1187846.zd2
0x1dc00060 : Vo DIST (v0.01), ./421894.zd2
0x1dc000d0 : Vo DarkRV (v0.01), ./450566.zd2
0x1dc00100 : VoDistFlt (v0.01), ./491526.zd2
0x1dc001d0 : Harmony (v0.01), ./524294.zd2
0x1dc00130 : PSOLAInst (v0.01), ./577542.zd2
0x1dc00120 : PSOLAInst (v0.01), ./614406.zd2
0x1dc00050 : Radio (v0.01), ./651270.zd2
0x1dc00198 : TEMPLATE4 (v0.01), ./675846.zd2
0x1dc00060 : Vo DIST (v0.01), ./1495046.zd2
0x1dc000d0 : Vo DarkRV (v0.01), ./1523718.zd2
0x1dc00100 : VoDistFlt (v0.01), ./1564678.zd2
0x1dc001d0 : Harmony (v0.01), ./1597446.zd2
0x1dc00090 : VoHdHall (v0.01), ./163846.zd2
0x1dc00130 : PSOLAInst (v0.01), ./1650694.zd2
0x1dc00120 : PSOLAInst (v0.01), ./1687558.zd2
0x1dc00050 : Radio (v0.01), ./1724422.zd2
0x1dc00198 : TEMPLATE4 (v0.01), ./1748998.zd2
0x1dc00090 : VoHdHall (v0.01), ./1236998.zd2
0x1dc000a0 : VoRoom (v0.01), ./1269766.zd2
0x1dc001c0 : TEMPLATE4 (v0.01), ./1290246.zd2
0x1dc001b8 : PSOLAInst (v0.01), ./1339398.zd2
0x1dc00020 : PitchDtct (v0.01), ./139270.zd2
0x1dc001b0 : PSOLAInst (v0.01), ./1404934.zd2
0x1dc000c0 : Vo Delay (v0.01), ./1441798.zd2
0x1dc00030 : 160 Comp (v0.01), ./1773574.zd2
0x1dc000b0 : PSOLAInst (v0.01), ./1810438.zd2
0x1dc00028 : VunVDtct (v0.01), ./1851398.zd2
0x1dc00170 : PSOLAInst (v0.01), ./1875974.zd2
0x1dc00160 : PSOLAInst (v0.01), ./1912838.zd2
0x1dc001a0 : Vo Robot (v0.01), ./1949702.zd2
0x1dc000a0 : VoRoom (v0.01), ./196614.zd2
0x1dc00020 : PitchDtct (v0.01), ./1212422.zd2
0x1dc00080 : Detune (v0.01), ./1470470.zd2
0x1dc00190 : VoTalkBox (v0.01), ./2023430.zd2
0x1dc00080 : Detune (v0.01), ./397318.zd2
0x1dc00030 : 160 Comp (v0.01), ./700422.zd2
0x1dc000b0 : PSOLAInst (v0.01), ./737286.zd2
0x1dc001d1 : Harmony (v0.01), ./73734.zd2
0x1dc00028 : VunVDtct (v0.01), ./778246.zd2
0x1dc00170 : PSOLAInst (v0.01), ./802822.zd2
0x1dc00160 : PSOLAInst (v0.01), ./839686.zd2
0x1dc00029 : VunVDtct (v0.01), ./86022.zd2
0x1dc001a0 : Vo Robot (v0.01), ./876550.zd2
0x1dc00190 : VoTalkBox (v0.01), ./950278.zd2
0x1dc00180 : VoVocoder (v0.01), ./2097158.zd2
0x1dc001c0 : TEMPLATE4 (v0.01), ./217094.zd2
0x1dc00030 : 160 Comp (v0.01), ./2207750.zd2
0x1dc001b8 : PSOLAInst (v0.01), ./266246.zd2
0x1dc001b0 : PSOLAInst (v0.01), ./331782.zd2
0x1dc000c0 : Vo Delay (v0.01), ./368646.zd2
1024006.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1110022.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1134598.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1146886.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1159174.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1212422.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1236998.zd2.txe1:This is a dense hall reverb.
1269766.zd2.txe1:This reverb effect simulates the acoustics of a room.
1339398.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
139270.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1404934.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1441798.zd2.txe1:This long delay has a maximum length of 4000 mS.
1495046.zd2.txe1:This compressor is in the style of the dbx 160A.
1523718.zd2.txe1:This simulates a plate reverb.
1564678.zd2.txe1:This compressor is in the style of the dbx 160A.
1597446.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
163846.zd2.txe1:This is a dense hall reverb.
1650694.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1687558.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1724422.zd2.txe1:This compressor is in the style of the dbx 160A.
1773574.zd2.txe1:This compressor is in the style of the dbx 160A.
1810438.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1851398.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1875974.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1912838.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
1949702.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
196614.zd2.txe1:This reverb effect simulates the acoustics of a room.
2023430.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
2097158.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
2207750.zd2.txe1:This compressor is in the style of the dbx 160A.
266246.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
331782.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
368646.zd2.txe1:This long delay has a maximum length of 4000 mS.
421894.zd2.txe1:This compressor is in the style of the dbx 160A.
450566.zd2.txe1:This simulates a plate reverb.
491526.zd2.txe1:This compressor is in the style of the dbx 160A.
524294.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
577542.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
614406.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
651270.zd2.txe1:This compressor is in the style of the dbx 160A.
700422.zd2.txe1:This compressor is in the style of the dbx 160A.
737286.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
73734.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
778246.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
802822.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
839686.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
86022.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
876550.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
950278.zd2.txe1:This is a simulation of the Demeter COMP-1 Compulator.
mungewell commented 2 years ago

Hmmm, seem to be some extra bytes (different amounts in each) in there for some reason...

    "ICON" / ICON,
    "TXJ1" / TXJ1,
    "TXE1" / TXE1,
    "INFO" / INFO,
    "DATA" / DATA,
    Padding(102),
    "PRMJ" / PRMJ,
    "PRME" / PRME,
)

$ python3 ~/zoom-zt2-sdw-github/decode_effect.py -d 950278.zd2
Container: 
    length = 120
    unknown = b'\x86P\xa4/\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'... (truncated, total 81)
    version = u'0.01' (total 4)
    group = 29
    id = 499122576
    name = u'VoTalkBox' (total 9)
    unknown2 = b'\x00' (total 1)
    groupname = u'VOCAL' (total 5)
    unknown3 = b"\x00\x00'Z\x96k\x16\x87\x00\x00\x00" (total 11)
    ICON = Container: 
        length = 182
        data = b'BM\xb6\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00(\x00'... (truncated, total 182)
    TXJ1 = Container: 
        length = 43
        data = b'Demeter COMP-1 C'... (truncated, total 43)
    TXE1 = Container: 
        length = 54
        description = u'This is a simulation of the Deme'... (truncated, total 54)
    INFO = Container: 
        length = 20
        data = b'\x00\x00\x00\x00VCLVOCAL\x00\x00\x00\x00'... (truncated, total 20)
    DATA = Container: 
        length = 71890
        data = b'\x7fELF\x01\x01\x01@\x00\x00\x00\x00\x00\x00\x00\x00'... (truncated, total 71890)
    PRMJ = Container: 
        length = 571
        data = b'{  \r\n\t"Parameter'... (truncated, total 571)
    PRME = Container: 
        length = 539
        xml = u'{  \r\n\t"Parameters":[  \r\n\t\t{  \r\n\t'... (truncated, total 539)
shooking commented 2 years ago

where are these V6 coming from? a G6?

mungewell commented 2 years ago

Err.... URL in the 1st post. ;-)

shooking commented 2 years ago

That is a strong hint. I didn't even know such a beast exists. Will read up about it. Thanks.

mungewell commented 2 years ago

Both the V3and V6 effects names.... number is offset in 136 file. V3_v1.10_Win_E/unzipped/.rsrc/1041/BIN/ZD2$ find . -name '*.ZD2' -exec bash -c "echo -n {}; python3 ~/zoom-zt2-sdw-github/decode_effect.py -d {} | grep ' name '" \; > v3_names.txt v3_names.txt

V6_v1.20_Win_E/unzipped/.rsrc/1041/BIN/ZD2$ find . -name '*.ZD2' -exec bash -c "echo -n {}; python3 ~/zoom-zt2-sdw-github/decode_effect.py -d {} | grep ' name '" \; > v6_names.txt v6_names.txt

mungewell commented 2 years ago

All effects in V3/V6 are in group 29, but some have variant groupname.

./217094.ZD2    group = 29,     groupname = u'VOCAL' (total 5)
./2207750.ZD2    group = 29,    groupname = u'DYNAMICS' (total 8)
./241670.ZD2    group = 29,    groupname = u'REVERB' (total 6)

Whereas other pedals use different group numbers.

./FDSPRING.ZD2    group = 9, groupname = u'REVERB' (total 6)
./FD_TWR1U.ZD2    group = 4, groupname = u'AMP' (total 3)
./GOLD_DRV.ZD2    group = 3, groupname = u'DRIVE' (total 5)
./GRAYCOMP.ZD2    group = 1, groupname = u'DYNAMICS' (total 8)
./GTGEQ71U.ZD2    group = 2, groupname = u'FILTER' (total 6)
mungewell commented 2 years ago

I worked around parsing effect(s) from the 136 file, but it looks like some have extra bytes in them For example `1597446.xml' has:

  ^M
        "Parameters":[  ^M
                {  ^M 
                   "name":"Gain",^M
                   "explanation":"Adjusts the gain.",^M 
                   "blackback":false,^M~L^A���^A
                   "pedal":false^M 
                },^M

Which is in hex:

00000000  7b 20 20 0d 0a 09 22 50  61 72 61 6d 65 74 65 72  |{  ..."Parameter|
00000010  73 22 3a 5b 20 20 0d 0a  09 09 7b 20 20 0d 0a 09  |s":[  ....{  ...|
00000020  09 20 20 20 22 6e 61 6d  65 22 3a 22 47 61 69 6e  |.   "name":"Gain|
00000030  22 2c 0d 0a 09 09 20 20  20 22 65 78 70 6c 61 6e  |",....   "explan|
00000040  61 74 69 6f 6e 22 3a 22  41 64 6a 75 73 74 73 20  |ation":"Adjusts |
00000050  74 68 65 20 67 61 69 6e  2e 22 2c 0d 0a 09 09 20  |the gain.",.... |
00000060  20 20 22 62 6c 61 63 6b  62 61 63 6b 22 3a 66 61  |  "blackback":fa|
00000070  6c 73 65 2c 0d 8c 01 ff  ff b6 01 0a 09 09 20 20  |lse,..........  |
                         ^^ ^^ ^^  ^^ ^^ ^^

And searching for this sequence I note that occurs only once, a little after the location of the particular ZD2.

$ python3 ~/SearchBin-github/searchbin.py -p 8c01ffffb601 136
Match at offset:        1646592       192000 in  136

I think that the ZD2 are encapsulated in some form of filesystem(??) within the 136 file. We will need to parse that before extracting the individual ZD2 files.

mungewell commented 2 years ago

So the data block starts at 0x0000F000 and there's a header which ends '0FFA' which is 4090 - this is probably a byte count..

My guess is that the ZD2 is split into 4090 byte blocks, with this header every 4096 bytes... should be easy enough to parse.

$ hexdump -C 136 | grep "ZDLF"
0000f000  ff ff 0b 00 fa 0f 5a 44  4c 46 78 00 00 00 1f 1c  |......ZDLFx.....|
00012000  ff ff 0e 00 fa 0f 5a 44  4c 46 78 00 00 00 0a ad  |......ZDLFx.....|
00015000  ff ff 11 00 fa 0f 5a 44  4c 46 78 00 00 00 93 2e  |......ZDLFx.....|
0001a000  ff ff 16 00 fa 0f 5a 44  4c 46 78 00 00 00 82 1b  |......ZDLFx.....|
shooking commented 2 years ago

Hi @mungewell Barsik documented this in his firmware decoder. You have been using MIDI to get the ZD2 but if you pull from the firmware you have to be aware of the block sizes etc. 4096 with 6 bytes in 3 x 2 bytes.

You can see here I used the truck whilst pulling out ZDLs from G5 and B1On. I guess if you don't pull the ZD2 out with the midi methods then you have to parse a bit.

https://github.com/shooking/ZoomPedalFun/blob/main/B1ON/DerivedData/DecodeFirmware/ProcessFirmware1.py

mungewell commented 2 years ago

Pushed a new tool to extract the effects. https://github.com/mungewell/zoom-zt2/blob/master/extract_effect_136.py

V6_v1.20_Win_E/unzipped/.rsrc/1041/BIN/extract$ python3 ~/zoom-zt2-sdw-github/extract_effect_136.py ../136

simon@thevoid:/media/simon/OS/Users/simon/Documents/ZoomFW/V6_v1.20_Win_E/unzipped/.rsrc/1041/BIN/extract$ ls
1024006.ZD2  1179654.ZD2  1376262.ZD2  1597446.ZD2  1851398.ZD2  217094.ZD2   241670.ZD2  491526.ZD2  737286.ZD2
106502.ZD2   1187846.ZD2  139270.ZD2   163846.ZD2   1875974.ZD2  2183174.ZT2  266246.ZD2  524294.ZD2  73734.ZD2
1097734.ZT2  1212422.ZD2  1404934.ZD2  1650694.ZD2  1912838.ZD2  2195462.ZT2  303110.ZD2  577542.ZD2  778246.ZD2
1110022.ZD2  1236998.ZD2  1441798.ZD2  1687558.ZD2  1949702.ZD2  2207750.ZD2  331782.ZD2  614406.ZD2  802822.ZD2
1134598.ZD2  1269766.ZD2  1470470.ZD2  1724422.ZD2  196614.ZD2   2244614.ZT2  368646.ZD2  61446.ZD2   839686.ZD2
1146886.ZD2  1290246.ZD2  1495046.ZD2  1748998.ZD2  2023430.ZD2  2256902.ZD2  397318.ZD2  651270.ZD2  86022.ZD2
114694.ZD2   1314822.ZD2  1523718.ZD2  1773574.ZD2  2097158.ZD2  2293766.ZT2  421894.ZD2  675846.ZD2  876550.ZD2
1159174.ZD2  1339398.ZD2  1564678.ZD2  1810438.ZD2  2170886.ZT2  2306054.ZT2  450566.ZD2  700422.ZD2  950278.ZD2

Not all 'files' are really ZD2, some look to be ZT2 which could be a way to associate ZD2 with the names that Zoom calls them...

mungewell commented 2 years ago

Yet to try these on a pedal, but making progress. Newly extracted ZD2 parse properly without the additional bytes.

Matching up id (in ZD2) to the filename (in ZT2) shows that there are multiple copies of effects in the 136 file. For example from the V3 unzipped EXE we get...

' 876550.ZD2.dump'                           id 499122592
1097734.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2170886.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2183174.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2195462.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2244614.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2293766.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
'1949702.ZD2.dump'                           id 499122592
1097734.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2170886.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2183174.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2195462.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2244614.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)
2293766.ZT2.dump-                    effect = u'V_ROBOT.ZD2' (total 11)

These ZD2(at least this one) are exactly the same

$ md5sum 876550.ZD2
55dade613d6f1a7135e86a135e6ee921  876550.ZD2
$ md5sum 1949702.ZD2
55dade613d6f1a7135e86a135e6ee921  1949702.ZD2
mungewell commented 2 years ago 
V3_v1.10_Win_E/unzipped/.rsrc/1041/BIN/ZD2_extracted$ ls
1097734.ZT2  2244614.ZT2  A_PSOLAI.ZD2  VOHDHALL.ZD2  VO_DEEP.ZD2   VO_HRMNY.ZD2  VUN_DTCT.ZD2
1773574.ZD2  2256902.ZD2  A_VHRMNY.ZD2  VOROOM.ZD2    VO_DELAY.ZD2  VO_OCTDN.ZD2  V_CORR_A.ZD2
2170886.ZT2  2293766.ZT2  A_VUN_DT.ZD2  VO_BEAT.ZD2   VO_DETUN.ZD2  VO_OCTUP.ZD2  V_CORR_N.ZD2
2183174.ZT2  2306054.ZT2  CMN_FFT.ZD2   VO_BRT_R.ZD2  VO_DIST.ZD2   VO_RADIO.ZD2  V_ROBOT.ZD2
2195462.ZT2  700422.ZD2   CMN_KLIB.ZD2  VO_CHILD.ZD2  VO_DRK_R.ZD2  VO_SYNTH.ZD2  V_TALK_B.ZD2
2207750.ZD2  A_PDTCT.ZD2  P_DTCT.ZD2    VO_CHO.ZD2    VO_DSFLT.ZD2  VO_UNISN.ZD2  V_VOCODE.ZD2
V6_v1.20_Win_E/unzipped/.rsrc/1041/BIN/extracted_zd2$ ls
1097734.ZT2  2244614.ZT2  A_PSOLAI.ZD2  VOHDHALL.ZD2  VO_DEEP.ZD2   VO_HRMNY.ZD2  VUN_DTCT.ZD2
1773574.ZD2  2256902.ZD2  A_VHRMNY.ZD2  VOROOM.ZD2    VO_DELAY.ZD2  VO_OCTDN.ZD2  V_CORR_A.ZD2
2170886.ZT2  2293766.ZT2  A_VUN_DT.ZD2  VO_BEAT.ZD2   VO_DETUN.ZD2  VO_OCTUP.ZD2  V_CORR_N.ZD2
2183174.ZT2  2306054.ZT2  CMN_FFT.ZD2   VO_BRT_R.ZD2  VO_DIST.ZD2   VO_RADIO.ZD2  V_ROBOT.ZD2
2195462.ZT2  700422.ZD2   CMN_KLIB.ZD2  VO_CHILD.ZD2  VO_DRK_R.ZD2  VO_SYNTH.ZD2  V_TALK_B.ZD2
2207750.ZD2  A_PDTCT.ZD2  P_DTCT.ZD2    VO_CHO.ZD2    VO_DSFLT.ZD2  VO_UNISN.ZD2  V_VOCODE.ZD2
mungewell commented 2 years ago

I tried to upload some of these to a G1FourX, mostly that failed.

It seems that most do not even show as files after an attempted upload - I'm guessing that the ZD2 is checked to ensure that all it's requirements (ie DLL) are met. It maybe that some of the unnamed (and 'CMN_*') are required to upload first.

The most simplest VOROOM.ZD2did upload/show in file listing, but was not visible as usable in the pedal. We may need to change the id/group/groupname/etc to be compliant with G1Four (and friends).

mungewell commented 2 years ago

So I took everything off my (broken screen) G1Four, and attempted to install all the V6 effects.... and they all were uploaded OK. All 36 are listed as files on the device and the FLST_SEQ contains them in group 29.

V6_v1.20_Win_E/unzipped/.rsrc/1041/BIN/extracted_zd2$ python3 ~/zoom-zt2-sdw-github/zoomzt2.py -I `ls *.ZD2`
Installing effect: 1773574.ZD2
Installing effect: 2207750.ZD2
Installing effect: 2256902.ZD2
Installing effect: 700422.ZD2
Installing effect: A_PDTCT.ZD2
Installing effect: A_PSOLAI.ZD2
Installing effect: A_VHRMNY.ZD2
Installing effect: A_VUN_DT.ZD2
Installing effect: CMN_FFT.ZD2
Installing effect: CMN_KLIB.ZD2
Installing effect: P_DTCT.ZD2
Installing effect: VOHDHALL.ZD2
Installing effect: VOROOM.ZD2
Installing effect: VO_BEAT.ZD2
Installing effect: VO_BRT_R.ZD2
Installing effect: VO_CHILD.ZD2
Installing effect: VO_CHO.ZD2
Installing effect: VO_DEEP.ZD2
Installing effect: VO_DELAY.ZD2
Installing effect: VO_DETUN.ZD2
Installing effect: VO_DIST.ZD2
Installing effect: VO_DRK_R.ZD2
Installing effect: VO_DSFLT.ZD2
Installing effect: VO_HRMNY.ZD2
Installing effect: VO_OCTDN.ZD2
Installing effect: VO_OCTUP.ZD2
Installing effect: VO_RADIO.ZD2
Installing effect: VO_SYNTH.ZD2
Installing effect: VO_UNISN.ZD2
Installing effect: VUN_DTCT.ZD2
Installing effect: V_CORR_A.ZD2
Installing effect: V_CORR_N.ZD2
Installing effect: V_ROBOT.ZD2
Installing effect: V_TALK_B.ZD2
Installing effect: V_VOCODE.ZD2

But I can't see the display to see if they are actually offered in the display. I'll try to build a patch manually.

mungewell commented 2 years ago

Some level of success squishing ZD2 files together.... "TEMPLATE4" is name hardcoded into VO_SYNTH.ZD2 Code block.

$ python3 decode_effect.py -o VO_SYMOD.ZD2 -D VO_SYNTH.ZD2 -C -T BOMBER.ZD2 $ python3 zoomzt2.py -I VO_SYMOD.ZD2 Installing effect: VO_SYMOD.ZD2

$ python3 zoomzt2.py -p 54 test.zptc $ python3 decode_preset.py -d test.zptc Container: fx_count = 5 name = u'Empty ' (total 10) ids = ListContainer: 117440528 0

IMG_20211109_210459508

mungewell commented 2 years ago

Well that's odd, after installing a modified effect.

Nov 9 21:43:44 thevoid pulseaudio[1323]: [pulseaudio] module.c: Failed to load module "module-alsa-card" (argument: "device_id="2" name="usb-ZOOM_Corporation_ZOOM_G_Series-00" card_name="alsa_card.usb-ZOOM_Corporation_ZOOM_G_Series-00" namereg_fail=false tsched=yes fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes card_properties="module-udev-detect.discovered=1""): initialization failed.

mungewell commented 2 years ago

I tried multiple ZD2 modules from the V6, it seems that when installed and enabled through being in an active patch the output audio is cut... once removed audio remains cut until pedal is rebooted.

There must be something different in the way the V3/V6 handle output audio. I note that the device supports USB audio, whereas the G1Four does not.

mungewell commented 2 years ago

$ python3 ../decode_effect.py -o VO_SYMOD.ZD2 -D VO_BEAT.ZD2 -C -T BOMBER.ZD2

Does NOT cut the audio, but does not seem to alter it either :-(

shooking commented 2 years ago

if you can send some commands I can try it on a G5n? Maybe with its audio interface it might behave differently?

mungewell commented 2 years ago

I checked in the code to 'merge' the ZD2's into the 'argparse' branch...

$ python3 ../decode_effect.py -o VO_SYMOD.ZD2 -D VO_BEAT.ZD2 -C -T BOMBER.ZD2

Will take the --donor-code and --donor-text sections from VO_BEAT.ZD2 and everything thing else from BOMBER.ZD2, and spit them out as VO_SYMOD.ZD2 which can then be uploaded the normal way. The new/merged effect will appears as BOMBER.ZD2 on the device, so you should de-install that first.

Audio cuts occur as soon as patch with modified ZD2 is selected.

I have also seen some combinations of ZD2's appear as-if effect does not exist (straight through line where Icon should be) but then lock up device, either switch patch and reboot, or hold 'mem' button when powering to revert to factory patches.

$ python3 decode_effect.py --help
usage: decode_effect [-h] [-d] [-s] [-b BITMAP] [-i INFO] [-c CODE] [-j]
                     [-x XML] [-t TEXT] [-D DONOR] [-B] [-T] [-I] [-C] [-X]
                     [-o OUTPUT]
                     FILE

positional arguments:
  FILE                  File to process

optional arguments:
  -h, --help            show this help message and exit
  -d, --dump            dump configuration to text
  -s, --summary         summarized configuration in human readable form
  -b BITMAP, --bitmap BITMAP
                        extract icon bitmap to FILE
  -i INFO, --info INFO  extract Info to FILE
  -c CODE, --code CODE  extract Code to FILE

Extract:
  Extract either English or Japanese segments of the ZD2

  -j, --japan           select Japanese version for export
  -x XML, --xml XML     extract XML to FILE
  -t TEXT, --text TEXT  extract Text to FILE

Donor:
  Take replacement sections from a donor

  -D DONOR, --donor DONOR
                        use sections from donor FILE
  -B, --donor-bitmap    extract icon bitmap from donor
  -T, --donor-text      extract Text from donor
  -I, --donor-info      extract Info from donor
  -C, --donor-code      extract Code from donor
  -X, --donor-xml       extracy XML from donor
  -o OUTPUT, --output OUTPUT
                        output combined result to FILE
mungewell commented 2 years ago

A little bit of self documentation.... noted that I should probably code to use FAT in 136 file (but that's not in the V3/V6 file).

Using the GCE FW:

$ mkdir extract_ZD2
$ cd extract_ZD2
$ python3 ~/zoom-zt2-sdw-github/extract_effect_136.py -s 77824  ../136
$ rm *.ZD2

$ find . -name "*.ZT2" -exec python3 ~/zoom-zt2-sdw-github/zoomzt2.py -b everything.zt2 {} \; >> unsorted.sh
$ cat unsorted.sh | sort | uniq > build_everything.sh

$ cp ~/zoom-zt2-sdw-github/empty.zt2 everything.zt2
$ ln -s ~/zoom-zt2-sdw-github/zoomzt2.py zoomzt2.py
$ bash build_everything.sh

$ python3 ~/zoom-zt2-sdw-github/extract_effect_136.py -s 77824 -z everything.zt2 ../136

$ find . -name "[0123456789]*.ZD2"
./160_COMP.ZD2
$ python3 zoomzt2.py -b everything.zt2 everything.zt2 | wc
    340    3400   25247
mungewell commented 2 years ago

@shooking

In processing the above, I ran in to the ZT2 limit for file size. Is this really needed?? Can the pedals cope with larger files?

ZT2 = Padded(8502, Sequence(
    "header" / Header,
    "groups" / GreedyRange(Group),
))
mungewell commented 2 years ago

Blending effects to upload them to a G1X_Four...

$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D VO_BEAT.ZD2 -C -T -X -I -B BOMBER.ZD2
$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D VO_SYNTH.ZD2 -C -T -X -I -B BOMBER.ZD2

result uploads, but does not seem to affect audio. Effect settings show "Template4"

$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D VO_HRMNY.ZD2 -C -T -X -I -B BOMBER.ZD2
$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D V_TALK_B.ZD2 -C -T -X -I -B BOMBER.ZD2

result uploads, but is shown as a 'bypass' (not missing efect icon). ToneLib confirms it is in patch....

$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D VO_DSFLT.ZD2 -C -T -X -I -B BOMBER.ZD2

result uploaded, but audio clicks then cuts when patch is selected. Audio remains cut until pedal is rebooted (on different preset). Preset parameters show as 'VoDistFlt' with 6 pages of parameters.

$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D 700422.ZD2 -C -T -X -I -B BOMBER.ZD2

result uploaded, but audio clicks then cuts when patch is selected. Audio remains cut until pedal is rebooted (on different preset). Preset parameters show as 'TONE+ZNR' with 5 pages of parameters.

$ python3 ../decode_effect.py -o V6_MOD1.ZD2 -D VOHDHALL.ZD2 -C -T -X -I -B BOMBER.ZD2

result uploaded, but audio clicks then cuts when patch is selected. Audio remains cut until pedal is rebooted (on different preset). Preset parameters show as 'VoHdHall' with 4 pages of parameters. Some parameters are not labelled.

Here's an example of what the 'get screens' midi reports: merged_VO_DSFLT.zip

mungewell commented 2 years ago

I started to wonder whether there is a requirement for more than one of the modules to be loaded at a time, after all the V3 and V6 have a rigid structure of audio flow.. V6_sound_flow . I looked the code sections, it looks like multiple ZD2 supply the same DLL name and also have some dependence on others. Dll.txt depends.txt

Perhaps we need to make a ZT2 and ZPTC to define which ZD2's are present, and how the are sequenced.

mungewell commented 2 years ago

Attempt to describe dependencies... V6_DLL_dependancy_tree

V6_DLL_dependancy_tree.pdf V6_DLL_dependancy_tree.ods

Note: Some of the DLL names are the same. I have drawn a box around them as I think only one of these can exist in a patch at one time.

mungewell commented 2 years ago

Well I seem to have 'killed' my G1_Four. This is the pedal with the broken screen, so I don't truly know what it's trying to tell me.... It does NOT boot properly, non on the LED flash and it doesn't present Midi. The 'reset to factory patches' trick does not appear to work.

It DOES enter FW flash mode and accept a FW image, but this does not improve things. It might be recoverable by accessing the SPI Flash directly, but for now is dead. :-(

I had replaced/merged the BDL with the code 700422.ZD2. On reboot this cuts audio, but merged VO_SYNTH.ZD2 did not produced audio.

I uploaded a merged CMN_KLIB.ZD2, which resulted in a audible 'ping'. After a reboot this is where the pedal fails to progress further.

It could be that the screen is showing a "preset error" or the like. As seen in #18

EDIT: I should add that simply having the original ZD2 files loaded into pedal did NOT cause the problem, I had to actively add then to the ZT2.

mungewell commented 2 years ago

With a fancy oscilloscope from work I am able to confirm that the pedal is reading the SPI EEPROM at boot time, and is (at some point) reading the FLST_SEQ that I wrote into the pedal. It is still in the state where it does not flash it's LEDs or present itself on the USB bus.

snooping_da_bus

If this is the cause of it not booting correctly, then it is logical that I should be able to overwrite the contents (changing 1 byte should be sufficient) and cause it NOT to see the 'badness'...

mungewell commented 2 years ago

Well magic is in the air.... whilst setting up with a BusPirate the pedal 'cured' itself and is now booting to show LEDs and present itself on Midi.

$ amidi -p hw:2,0,0 -S 'F0 7e 00 06 01 F7' -r temp.bin -t 1 ; hexdump -C temp.bin | head
15 bytes read
00000000  f0 7e 00 06 02 52 6e 00  0c 00 32 2e 30 30 f7     |.~...Rn...2.00.|
0000000f

It's internal drive is COMPLETELY empty, except for GUARD.ZT2 and FLST_SEQ.ZT2 (which only holds only BYPASS.ZD2). All of the .raw sound files are missing and there is no .BDL file.... perhaps this is what the eeprom holds before it is factory configured.

I don't remember if we have a way to read the device's serial number?

nomadbyte commented 1 year ago

Not sure if this quest got any way practical but adding here a few findings.

It seems that the "160 Comp" effect is a ZNR (noise reduction) BDL module. So it's likely to be named VO_TONEZ.BDL (type/fxid: 0x1DC00030, md5sum: d309d365adc6bb834194afeb18071497). It's not listed in the FLST_SEQ file, but is referenced as ZNR+TONE.

Below is a list of the V6-SP modules from the fw v1.20. All of the modules are of v0.01 and appear to have ZD2 structure. Even though there's some variation in category names (mostly VOCAL, a few DYNMCS, REVERB, and VO), all of them have the same category-id: 0x1D = 29. Perhaps, category wasn't yet a feature in V6.

type file md5sum
0x1DC00030 __VO_TONEZ.BDL__ d309d365adc6bb834194afeb18071497
0x1DC00011 A_PSOLAI.ZD2 8c2a3ef1859bc1db7c2779abc1027845
0x1DC00020 P_DTCT.ZD2 9025c974975b1b2cf5ff3e228fccb5a4
0x1DC00021 A_PDTCT.ZD2 999ce07328339c96d3a92a41c8e0c5c6
0x1DC00028 VUN_DTCT.ZD2 cfeaed72bdf9112f7fa53af19e5176d7
0x1DC00029 A_VUN_DT.ZD2 f8fb3c3d9717c4b5847892d159b0e562
0x1DC00050 VO_RADIO.ZD2 2bbea112cf1641699df06aff0d9f946a
0x1DC00060 VO_DIST.ZD2 e8b9dceb39e86defd9cf7f97a3f3c790
0x1DC00070 VO_CHO.ZD2 d8f269af7cee211d309f474f630b8560
0x1DC00080 VO_DETUN.ZD2 3b63e35bd329dbe88332d6526c5c7625
0x1DC00090 VOHDHALL.ZD2 70b0fc891b8fea90b8e6aeabbb77e891
0x1DC000A0 VOROOM.ZD2 82802da75b39301f3f1d6dedc730efcc
0x1DC000B0 VO_UNISN.ZD2 1ce0a7363a97a113e95313500c0728c8
0x1DC000C0 VO_DELAY.ZD2 43889cb409f5e8284caa359161775378
0x1DC000D0 VO_DRK_R.ZD2 aa73038283999133f1fbca8098cd0c43
0x1DC000E0 VO_BRT_R.ZD2 a82e93e24f390b5f81ef05433973dde4
0x1DC00100 VO_DSFLT.ZD2 fd7284fbf0f9681e5f60f63b723ec4f0
0x1DC00120 VO_OCTUP.ZD2 7061c29b98d32ef8aa88d51377c0097f
0x1DC00130 VO_OCTDN.ZD2 797be77e43ce8a17b8d50cb42deae59b
0x1DC00160 V_CORR_N.ZD2 5d501fe9aa1e17bcbc7fc10653dc9dd5
0x1DC00170 V_CORR_A.ZD2 d063e6b470207c7c030bcc7e545903ab
0x1DC00180 V_VOCODE.ZD2 a86202852828f9636f63d868041ee4b2
0x1DC00190 V_TALK_B.ZD2 e7ca48eb2c15d5c8bc74c525b86c7fd1
0x1DC00198 VO_SYNTH.ZD2 ce96084f2feed07dfa9b45cdd8985265
0x1DC001A0 V_ROBOT.ZD2 55dade613d6f1a7135e86a135e6ee921
0x1DC001B0 VO_DEEP.ZD2 8a4331a7b26a451cf39804072164aa45
0x1DC001B8 VO_CHILD.ZD2 d05826c9f1e9a6a9ce8b01dae0aa2e13
0x1DC001C0 VO_BEAT.ZD2 ea60a25331ce52127058eaaa6ec86cd3
0x1DC001D0 VO_HRMNY.ZD2 be678a50c139e757b284fece87bb613d
0x1DC001D1 A_VHRMNY.ZD2 5d501fe9aa1e17bcbc7fc10653dc9dd5
0x1DC0FFFE CMN_KLIB.ZD2 2681063e0249fe3a42820dc37a2d6873
0x1DC0FFFF CMN_FFT.ZD2 18f283bd6bfb22a34a8680e919434c9d
nomadbyte commented 1 year ago

I wonder what's the purpose of the BDL modules in these effect processors? I understand that G-models have G_OUT_EQ.BDL, B-models have B_OUT_EQ.BDL, and now it appears that V6 has a VO_TONEZ.BDL.

I guess all of the BDL modules are in fact of ZD2 type and drive some part of the internal processing chain.

mungewell commented 1 year ago

My assumption is that the BDL file is automatically loaded into the output stage of the device, and that the G1 and B1 are different to allow different frequencies for the EQ. It may be that there is some special linkage to the controls, or maybe whatever controls are just displayed at the appropriate stage of the menu.

We could experiment by making them a normal ZD2 effect and see if they can be loaded into a normal slot.

WRT 'category' the V6 does not have a display, as with the AC-2 and AC-6 the effects are assigned in groups to knobs which can be turned to select.

nomadbyte commented 1 year ago

With G1 FOUR (v2.00) the G_OUT_EQ.BDL module appears to also have a category-id:0x1D = 29, same with B_OUT_EQ.BDL for B1 FOUR. What's curious is that internally it references the category as 29.BUILT_IN, just as all of the V6-SP modules.

So this gives all reasons to assign category-id: 29 = BUILT_IN.

type file md5sum
0x1D100010 G_OUT_EQ.BDL(v1.0) bbea3c08c0d5bb616f2731052101c67a
0x1D100020 B_OUT_EQ.BDL(v1.0) 044c648c200bdd57b1ca2d1b3164d61a
mungewell commented 1 year ago

I guess that you have found the one from the B1Four FW v1.10...

B1_FOUR_v1.10_Win_E/others$ python3 ~/zoom-zt2-sdw-github/decode_effect.py --summary -m B_OUT_EQ.BDL 
0x1d100020 : OutputEQ (v1.00, 3.94%), 0x044c648c200bdd57b1ca2d1b3164d61a, B_OUT_EQ.BDL

This one has the following:

    version = u'1.00' (total 4)
    group = 29
    id = 487587872
    aname = u'OutputEQ' (total 8)
    bname = b'OutputEQ\x00.0' (total 11)
    name = u'OutputEQ' (total 8)
    groupname = u'EQ' (total 2)

I have no objection to define this group as "BUILT_IN"

mungewell commented 1 year ago

So the reference to 'BUILT_IN' is from doing strings on the code segment?? It looks like this is a location on the compile machine's drive... for the AC-3 effects this is actually

C:\Project\ZDL_Ver2.00\PROGRAM\Gen01\29.AG_EFX\ZDL_AG_EFX_AgComp\Debug

And doesn't match the groupname which is present in the header the ZD2.

nomadbyte commented 1 year ago

It's a question of context supported by this utility. In the context of "G Series" devices the effects in category 0x1D are only the .BDL modules (G_OUT_EQ.BDL, B_OUT_EQ.BDL). These modules reference the category as EQ in the ZD2 header, while internally these modules are built in 29.BUILT_IN directory (it appears that the naming convention is <cat-id>.<cat-short-name>).

The V6 and AC-3 line of the devices are somewhat different. All of modules for these devices appear to be in the 0x1D category, and these don't have a distinct category taxonomy, V6-SP references its category as VOCAL, AC-3 references it as AG.EFX in ZD2 header, and internally those were built as 29.BUILT_IN and 29.AG_EFX resp.

So, given the "G Series" being more of the focus for this utility, the catid 29 could be named either EQ or BUILT_IN. The latter, in my opinion, corresponds more to the intended use (these BDL modules are rather for internal use). Indirectly this also covers the differences in nomenclature in V and AC device lines.

nomadbyte commented 1 year ago

@mungewell: Well magic is in the air....

Revisiting the thriller... So did you manage to revive your G1 FOUR to functional condition after that episode, when it got stuck failing to boot and then ??somehow ended up with its flash drive completely empty?

nomadbyte commented 1 year ago

I have a long story to tell but will shorten it to just a single observation -- looks like the G1 FOUR Firmware Updater does not actually overwrite the filesystem part of the on-device flash storage, despite including its size in the total bytes written count.

The Firmware update process overwrites Presets, Boot, and Main parts (~750kB overall) but apparently skips the filesystem, which actually contains the ZD2 modules and FLST_SEQ.ZT2. Not sure if it has always been the case or it's something specific to v2.0.

This may explain why @mungewell was unable to restore the pedal to its functional state after it got "killed" (I guess, failed to boot, stuck at ZOOM logo) after the experimentation with V6-SP originated modules.

Not sure how specifically he was able to make it unstuck (which likely had to clear the filesystem contents) but he mentioned that the Firmware Updater was not much of use in that case, even though it was completing the FW flashing successfully. Well, indeed, only the service part of the firmware was being re-flashed (via Sysex) but the lock up was due to the errant modules present on the filesystem, thus it persisted.

shooking commented 1 year ago

The GCE-3 seems to send firmware, then ZD2s then patches. My guess is a Zoom updater for a normal pedal does the same - with exception of the model (my guess based on taking b1 and g1 apart is the model is encoded at factory on a chip, someone magic markers the model on the board, it is forever a G1 or B1 ... until someone cleverer than me updates a chip and now its another model).

I already demoed using @mungewell code to turn a g1x into b1x - I didn't touch the BDL file. It seemed to behave like a b1x until on tries the firmware update.

Anyhow, glad to have another pair of eyes looking at this stuff. I am close to completing hardware breakout of B1XFour. Some code then dilemma of where to drill holes for switches in new case.

mungewell commented 1 year ago

As noted on Nov 15, 2021 my broken pedal magically fixed itself when I was poking around connecting a BuPirate to SPI memory. This fits with the suggestion that the installer code does not re-flash the 'drive'... however I may have triggered a reflash by effectively corrupting (at read time) the internal drive with my probes.

Meaning that the 'broken' pedal's drive was NOT corrupt, but caught in a crash/boot loop due to the ZD2/BDL changes I had installed.

Way back with hacking Linksys routers this was a known technique - short flash address pins to corrupt checksum which caused device to enter boot load mode, where you could upload new code....

nomadbyte commented 1 year ago

Did you eventually re-populate the emptied filesystem by hand on your "resurrected" G1 FOUR? That is you did you use your scripts or you re-flashed it again using the Firmware Updater .exe to get all the BDL/ZD2/ZT2 contents back onto the internal filesystem?

P.S. Did you happen to snap some pics of the "flash address pins" you mentioned, just for the reference?

nomadbyte commented 1 year ago

FWIW: In Firmware Update mode, the device does not communicate much Sysex back to the host, it does send back the usual ID info, then simply OK's whatever chunks sent by the host.

Only a single Sysex request <60 30> seems to return a <60 04 30> reply with something resembling a checksum. Given that <60> apparently marks the filesystem-related commands, the checksum indeed may be for the filesystem.

However, I don't see how this may relay the state of the actual content for the purposes of deciding whether to overwrite it. Even in the course of the usual ZD2 module install/uninstall using GL or ToneLib the contents of the filesystem will change, thus its checksum, which should then allow the overwriting. Otherwise, there is no practical way to re-flash even an empty but "healthy" filesystem using the Firmware Updater.

@mungewell , in the course of your V6-SP experimentation, you said you pretty much wiped the the original filesystem contents and filled it with the V6-SP ZD2 modules. This should've changed the contents checksum (if such indeed is there), yet the Firmware Updater would not overwrite the contents, as you observed.