search

mungewell / zoom-zt2

Python script to install/remove effects from the Zoom G1Four pedal
MIT License
62 stars 11 forks source link

Meaning of the last 16 bytes at end of ZD2 file #49

Open mungewell opened 2 years ago

mungewell commented 2 years ago

@nomadbyte asked on #48

seems that there is duplication of these byte sequences, which some being especially "popular", this suggest that they likely have some encoding in them.

zoom_fx_AllZDL7$ find -name "*.dump" -exec grep -A 1 -H hex5 {} \; | cut -d '/' -f 4- | grep "0000" | cut -d '-' -f 2-| sort | uniq -c
      1     0000   01 58 EC 97 D5 0D 7C 15 08 AA 48 0F 41 C8 D0 14   .X....|...H.A...
      1     0000   02 3B 8F CA E4 E1 F9 B9 D1 2D 22 56 EE 3C 03 CB   .;.......-"V.<..
     86     0000   02 D7 E8 39 2C 53 CB C9 12 1E 33 74 9E 0C F4 D5   ...9,S....3t....
      1     0000   08 68 51 AD C0 C5 EC 02 94 12 EC 94 69 30 F6 83   .hQ.........i0..
      1     0000   0A 8B 6B D8 D9 B0 8B 08 D6 4E 32 D1 81 77 77 FB   ..k......N2..ww.
     13     0000   0D 92 29 1D 86 29 99 72 DB 74 1C FA 4F 37 B8 B5   ..)..).r.t..O7..
     57     0000   0E 5F 50 00 D4 61 8D BE 7B 05 15 07 3B 33 82 1F   ._P..a..{...;3..
      1     0000   0E 6A 0A 42 C4 71 BE 0F E3 C3 01 DB FF 30 B2 1A   .j.B.q.......0..
      6     0000   0F 14 05 65 1B 28 61 C9 C5 E7 2C 8E 46 36 08 DC   ...e.(a...,.F6..
      4     0000   10 50 9B C8 81 43 29 28 8A F6 E9 9E 47 A1 81 48   .P...C)(....G..H
      1     0000   12 26 7F EC AE FC 45 01 74 D7 6D 9F 9A A7 75 5A   .&...E.t.m...uZ
     22     0000   15 1D 9A 95 C1 9B E1 C0 7E E9 A8 9A A7 86 C2 B5   ........~.......
     45     0000   18 70 92 DA 64 54 CE B1 85 3E 69 15 F8 46 6A 04   .p..dT...>i..Fj.
      1     0000   1B 74 5C FE 77 9D 6B 6E 93 48 A7 44 B3 3E 02 B8   .t\.w.kn.H.D.>..
      1     0000   1C 60 CA 20 CF 5D D6 2F 45 53 29 D7 A8 59 CC 0D   .`. .]./ES)..Y..
     93     0000   20 9A 50 EE 40 78 36 FD 12 49 32 F6 9E 7D 49 DC    .P.@x6..I2..}I.
      1     0000   26 09 7E F6 7A D8 ED 55 97 12 CA 57 02 B6 6C 08   &.~.z..U...W..l.
    732     0000   29 23 BE 84 E1 6C D6 AE 52 90 49 F1 F1 BB E9 EB   )#...l..R.I.....
     89     0000   2C 80 7E C5 99 D5 E9 80 B2 EA C9 CC 53 BF 67 D6   ,.~.........S.g.
      1     0000   2D 62 28 D3 50 10 17 5D E7 E8 F7 C4 E2 A8 E1 C8   -b(.P..]........
      1     0000   30 CD 90 A9 A5 87 4B F4 8E AF 70 EE A3 A6 2A 25   0.....K...p...*%
      3     0000   31 6C CD A4 9E DE 81 A3 8C 98 10 FF 9A 43 CD CF   1l...........C..
      5     0000   32 34 09 2C 4D 01 8F 17 56 C6 DB 9D C8 A6 D8 0B   24.,M...V.......
      1     0000   32 B9 F4 04 02 2D 2C E0 1E 01 74 F8 43 90 0B A2   2....-,...t.C...
      1     0000   34 4E 60 3C E5 DE 02 98 42 B2 0D 2B B6 14 EC BB   4N`<....B..+....
      1     0000   36 D2 34 9E 23 F6 44 39 2C 8E A9 C4 9D 40 C1 32   6.4.#.D9,....@.2
      1     0000   37 16 92 08 53 1C B5 37 3E 37 0E 72 FB F2 46 CC   7...S..7>7.r..F.
      1     0000   40 5B A3 33 B9 35 88 68 AD E1 2A D5 B2 32 5D 0A   @[.3.5.h..*..2].
      1     0000   42 E9 3F 8B 31 4F B1 8A 77 F7 90 AE 04 9F ED D6   B.?.1O..w.......
     69     0000   48 D3 8F 75 E6 D9 1D 2A E5 C0 F7 2B 78 81 87 44   H..u...*...+x..D
      1     0000   4B 41 E5 92 28 3F 62 F5 B8 AE FD A9 49 2B 31 51   KA..(?b.....I+1Q
      1     0000   54 4D 49 CD 49 72 0E 21 9D BF 8B BE D3 39 04 E1   TMI.Ir.!.....9..
     28     0000   54 BF 9A E7 D9 23 D1 55 90 38 28 D1 D9 6C A1 66   T....#.U.8(..l.f
      1     0000   57 C7 50 59 BF BD 1C 27 03 28 7F 5D 89 5F B9 49   W.PY...'.(]._.I
      1     0000   58 3F 74 47 EF 42 54 2E DF BE 39 B6 E6 0C 67 1C   X?tG.BT...9...g.
      1     0000   5A 9D 2E 9C 64 87 98 27 A8 48 EE 98 33 48 48 BD   Z...d..'.H..3HH.
      8     0000   5A BE 65 27 2A F6 07 AD A1 BE 65 A6 B4 C9 C0 69   Z.e'*.....e....i
      2     0000   5B 62 8A 8A 8F 27 5C F7 E5 87 4A 3B 32 9B 61 40   [b...'\...J;2.a@
      1     0000   5C 4E FA 9F 21 FC 3C 36 85 8E 81 B0 7D BF EE B1   \N..!.<6....}...
    102     0000   5D D4 CB FC 96 F5 45 3B 13 0D 89 0A 1C DB AE 32   ].....E;.......2
      6     0000   5E 4C EC 03 4C 73 E6 05 B4 31 0E AA AD CF D5 B0   ^L..Ls...1......
     15     0000   5E 4E E1 30 9C FE D9 71 9F E2 A5 E2 0C 9B B4 47   ^N.0...q.......G
      1     0000   5E 6A 40 FC 80 76 02 4B 17 6B 36 B1 21 DB 7D 5A   ^j@..v.K.k6.!.}Z
      1     0000   5F C5 17 C5 3E FC 33 63 C3 84 92 AB 08 A3 AA 3F   _...>.3c.......?
     80     0000   61 CD D1 1E 9D 9C 16 72 72 E6 1D F0 84 4F 4A 77   a......rr....OJw
      1     0000   64 B8 33 8B 9A D9 C3 40 8E 06 61 37 74 32 83 20   d.3....@..a7t2. 
      6     0000   65 38 2A 46 89 A9 82 79 7A 76 78 C2 63 B1 26 DF   e8*F...yzvx.c.&.
      6     0000   6D 0E E3 6C 28 A1 1E 20 1D CB C2 03 3F 41 07 84   m..l(.. ....?A..
      1     0000   6E DE F1 C2 A1 C3 0E 49 B5 C7 80 BC 3E C2 0B E2   n......I....>...
      1     0000   71 AF F2 64 D0 F2 48 41 D6 46 5F 09 96 FF 84 E6   q..d..HA.F_.....
     34     0000   76 2D D0 C2 C9 CD 68 D4 49 6A 79 25 08 61 40 14   v-....h.Ijy%.a@.
      6     0000   7F 70 F0 A7 54 86 32 95 AA 5B 68 13 0B E6 FC F5   p..T.2..[h.....
      3     0000   84 C6 C3 B1 A7 30 4A 10 EE 75 6F 03 2F 9E 6A EF   .....0J..uo./.j.
      6     0000   87 78 73 4D D0 BE 82 BE DB C2 46 41 2B 8C FA 30   .xsM......FA+..0
      7     0000   88 81 38 61 6B 68 12 62 F9 54 D0 E7 71 17 48 78   ..8akh.b.T..q.Hx
      1     0000   89 F8 65 22 51 3C 05 3D 4A B3 BE FD 8D 9E 19 D0   ..e"Q<.=J.......
      1     0000   8D AA 17 B1 AE 05 05 29 C6 82 7F 28 C0 EF 6A 12   .......)..(..j.
      8     0000   8F 73 53 40 43 8A AF 7E CA 6F D5 CF D3 A1 95 CE   .sS@C..~.o......
     37     0000   96 73 0E D9 16 2F 67 68 D4 F7 4A 4A D0 57 68 76   .s.../gh..JJ.Whv
      6     0000   99 CD D3 0D F0 44 3A B4 A6 66 53 33 0B CB A1 10   .....D:..fS3....
      6     0000   99 E7 3D B1 60 DE B1 80 AD 08 41 E9 67 41 A5 D5   ..=.`.....A.gA..
      8     0000   9F E4 18 9F 15 42 00 26 FE 4C D1 21 04 93 2F B3   .....B.&.L.!../.
      1     0000   A1 FE 69 D4 FA 04 67 ED BF C9 19 14 D1 3F F8 F2   ..i...g......?..
      8     0000   A5 F6 79 BD F7 EB 15 B8 E0 E1 60 8F 6E 3C 7B F4   ..y.......`.n<{.
     94     0000   AD 4F 14 F2 44 40 66 D0 6B C4 30 B7 32 3B A1 22   .O..D@f.k.0.2;."
      1     0000   AF 63 95 41 E3 BD DD A1 58 DA 4F 31 38 CE FD CA   .c.A....X.O18...
      8     0000   B0 95 57 F5 DF 80 6C 6D 8D 74 D9 8B 43 65 11 08   ..W...lm.t..Ce..
     33     0000   B1 3B 6A A5 11 28 C1 8C D6 A9 0B 87 97 8C 2F F1   .;j..(......../.
    125     0000   B3 12 4D C8 43 BB 8B A6 1F 03 5A 7D 09 38 25 1F   ..M.C.....Z}.8%.
    113     0000   B3 A6 DB 3C 87 0C 3E 99 24 5E 0D 1C 06 B7 47 DE   ...<..>.$^....G.
      1     0000   B7 26 B6 A2 78 D8 02 D1 EE AF 13 21 BA 59 29 DE   .&..x......!.Y).
      1     0000   B8 2F 73 E2 51 7E 7D 1D D8 84 D3 1F 01 BE 50 6B   ./s.Q~}.......Pk
      1     0000   B9 CE B7 B1 11 B3 76 64 69 A3 08 CD 15 28 1C EB   ......vdi....(..
      1     0000   BC F0 CA 0E 3D 6D 94 31 92 74 AF 8D B5 A4 90 D5   ....=m.1.t......
      1     0000   BE 00 56 2F 87 84 6A 54 36 5F 6C DA 3C C5 0A CB   ..V/..jT6_l.<...
     82     0000   BF 14 D6 7E 2D DC 8E 66 83 EF 57 49 61 FF 69 8F   ...~-..f..WIa.i.
      1     0000   C0 A6 61 1D 92 E5 C8 BC 78 2B 4A 77 C3 18 59 0A   ..a.....x+Jw..Y.
      1     0000   C3 8F D6 2A 21 14 A9 D4 04 11 01 18 8D AE BB 73   ...*!..........s
      1     0000   C4 67 BC 2F F2 FF C9 CC 0F 13 90 A9 24 11 1E AD   .g./........$...
      1     0000   C5 5C EF 65 E5 69 C4 30 DE AF 9C E2 5E B6 36 4B   .\.e.i.0....^.6K
     10     0000   C6 A6 2A 65 CF F4 47 36 21 EB DD 4B A2 6A 99 A8   ..*e..G6!..K.j..
      6     0000   CA 27 FF D8 9D 14 4D F4 79 27 59 42 7C 9C C1 F8   .'....M.y'YB|...
      1     0000   CA 80 EC 2E 45 1A 39 35 AA BE 1C 65 CE 77 BD 57   ....E.95...e.w.W
      6     0000   CA BE 7D 9F 89 8A 41 1B FD B8 4F 68 F6 72 7B 14   ..}...A...Oh.r{.
      1     0000   CB 1D 36 E9 49 74 02 55 D2 AC 1A 0B F7 A9 26 23   ..6.It.U......&#
      5     0000   CD 8C 87 20 23 64 B8 A6 87 95 4C B0 5A 8D 4E 2D   ... #d....L.Z.N-
      1     0000   CF 3A 65 CA 58 2C 2D E2 0C 60 DC 2C 62 05 3C 62   .:e.X,-..`.,b.<b
     50     0000   D4 9F D4 A4 59 7E 35 CF 32 22 F4 CC CF D3 90 2D   ....Y~5.2".....-
      6     0000   DA 29 6D 3E 62 E0 96 12 34 BF 39 A6 3F 89 5E F1   .)m>b...4.9.?.^.
      1     0000   E4 85 E9 12 07 8B C6 EC 66 E2 A3 F3 B9 F4 90 06   ........f.......
      1     0000   E5 5A DC E9 77 5D EB B5 69 C5 3A 6C 93 98 0D 57   .Z..w]..i.:l...W
      1     0000   E9 5A B8 BF B7 55 DA 5C 13 EA 60 3E 59 95 25 DC   .Z...U.\..`>Y.%.
      1     0000   EA 26 ED 55 4E 80 84 D9 2B F8 37 B8 ED D5 7A A0   .&.UN...+.7...z.
      1     0000   EA 72 1E 82 8D 71 A8 8C B8 5E D9 4E AF FA BF B0   .r...q...^.N....
      1     0000   EB 87 9A DF 04 68 B2 A2 D5 E6 A4 C6 BC 77 5F 8D   .....h.......w_.
      1     0000   F2 AF 6D 80 D2 E5 83 57 45 E9 A0 C4 EC 67 0F FA   ..m....WE....g..
      6     0000   F3 A8 8D FE BE F2 EB 71 FF A0 D0 3B 75 06 8C 7E   .......q...;u..~
     42     0000   F4 45 D3 D8 28 CE 0B F5 C5 60 59 3D 97 27 8A 59   .E..(....`Y=.'.Y
    107     0000   F6 22 91 9D E1 8B 1F DA B0 CA 99 02 B9 72 9D 49   ."...........r.I
     44     0000   FA 16 BB 11 AD AE 24 88 79 FE 52 DB 25 43 E5 3C   ......$.y.R.%C.<
      1     0000   FD 40 A4 1D 37 0A 1F 65 74 50 95 68 7D 47 BA 1D   .@..7..etP.h}G..

The quoted sequence from #48 is actually 17 bytes, it really starts with 29 23 BE... which super popular with 732 occurrences in my test set.

Next is to figure out what those effects have in common....

mungewell commented 2 years ago

The is only 1 (current) effect that matches the same value as AIR (v1.3)

zoom_fx_AllZDL7$ find -name "*.dump" -exec grep -A 1 -H "5F C5 17 C5 3E FC 33 63 C3 84 92 AB 08 A3 AA 3F" {} \; | grep "0000"     
./B6/unzipped/SPRING88.ZD2.dump:    0000   5F C5 17 C5 3E FC 33 63 C3 84 92 AB 08 A3 AA 3F   _...>.3c.......?
AllZDL7$ grep SPRING ./B6/unzipped/list_sorted.txt 
0x0a090050 : Spring (v1.00, 13.07%), 0xd45faf8782d9348df71947d6cd3cf141, ./B6/unzipped/SPRING88.ZD2
mungewell commented 2 years ago

For reference Air (V1.3) was in the G3n FW. 0x09000010 : Air (v1.30), 0xd340064af8e35a99133c72acc790b61b, ./G3n_v2.20_Win_E/effects/AIR.ZD2

nomadbyte commented 2 years ago

Given there are 16 bytes, could these represent a MD5 checksum of something?

Notable that for "AIR.ZD2" versions 1.40 and 1.50 both have the same trailing 16-bytes. The only other differing parts are the checksum (#48), the bit-field after the checksum, and the version details (number and some additional bytes).

mungewell commented 2 years ago

These bytes are the same for many different effects (uniq -c count above).

There is some duplication in my dataset as effects are present on multiple pedals.... but these counts would be much lower count if these bytes were a checksum.