How to find G1XFour drum pattern data and encode it

[nopeg]

As I understand decode_pattern finds offset from the bitmap of this header at which actual pattern data is stored

I understand that this script is just a parser that uses construct to decode pattern data from the memory pointed to

The binary search confirmed that the location of the data is at the same offset (I don't know what is the next step)

I need help with understanding some basics of binary search and decoding all this stuff, like how to use decoded patterns to see how this system works.

Sorry in advance for my incompetence

[mungewell]

@nopeg Correct, the script just parses the binary data - the format is guessed from staring at the hexdump and figuring out what it might mean. The binary search I did was just to help you locate the data in the G1X_Four FW, which turned out to the be the same as G1_Four.

The bytes that you highlight in your image are suspected to be a 3 byte pointer to the data which defines how the pattern plays the samples: https://github.com/mungewell/zoom-zt2/blob/master/decode_patterns.py#L29

    "pointer" / Int24ul,
    Const(b"\xC0"),
)

The script has the -P option which dumps these pointers to the terminal

simon@the-void:/media/simon/OS/Users/simon/Documents/ZoomFW/G1_FOUR_v2.00$ python3 ~/zoom-zt2-sdw-github/decode_patterns.py -T 407304 -D 457078 -P ../G1X_FOUR_v2.00/unzipped/.rsrc/1041/BIN/129
NewWave    0x0006F976 
R&B1       0x0006FF76 
PopRock    0x000700E6 
Samba2     0x0007019E 
Disco      0x00070306 
MtlCore    0x000703B6 
16Beats3   0x000705B6 
Pop        0x0007065E 
Samba1     0x00070706
...

The actual value of the pointer is uncertain. I suspect that it is the memory location of data after the file is loaded - rather than the offset of data within the file. It seems that the deltas between the pointers match the file data, so I use this...

https://github.com/mungewell/zoom-zt2/blob/master/decode_patterns.py#L116

        pointer = int(item['pointer']) - first + int(options.drums)

So a little note/clue to help you understand 'construct' better. The GreedyRange() creates an array of the sub-item, until the binary data fails to parse (ie data does not match a Const() or some other def). In this way the script can figure out how many patterns there are and split them appropriately into items.

If you use the -d / --dump option you can see how the data has been parsed....

simon@the-void:/media/simon/OS/Users/simon/Documents/ZoomFW/G1_FOUR_v2.00$ python3 ~/zoom-zt2-sdw-github/decode_patterns.py -T 407304 -D 457078 -d ../G1X_FOUR_v2.00/unzipped/.rsrc/1041/BIN/129 | head -n 50
Container: 
    items = ListContainer: 
        Container: 
            name = 'Dummy' (total 5)
            sample1 = '' (total 0)
            sample2 = '' (total 0)
            sample3 = '' (total 0)
            sample4 = '' (total 0)
            sample5 = '' (total 0)
            sample6 = '' (total 0)
            sample7 = '' (total 0)
            sample8 = '' (total 0)
            sample9 = 'Click.raw' (total 9)
            tsig_top = 4
            tsig_bot = 4
            bars = 2
            pointer = 1578848
        Container: 
            name = 'GUIDE' (total 5)
            sample1 = 'Kik1.raw' (total 8)
            sample2 = 'SnrAmb.raw' (total 10)
            sample3 = 'Snr1.raw' (total 8)
...

[mungewell]

If you wanted to create your own patterns, I guess you have to answer whether the understanding of the formatting is correct and whether it's possible to change it within the 129 file... which could then be repacked into the FW EXE file.

Hint: Found this tool, which was able to replace the '133' file in the exe. http://www.angusj.com/resourcehacker/

I suspect that you'd have to work within the space already used/allocated in the file, as to not overwrite/displace data after the drum data. Maybe a first test would be to swap over the pointers and/or drum data to confirm that you can actually create a FW image and hear the effects of a change.

This first step(s) might just be with a hex-editor and manually changing data. The script/Construct parse could then be used to automate the process.

Q2. Does the pedal freak out when there are less (or more) items in the 'table' (with names/samples)?

[mungewell]

I have a nagging memory that the data for the drums may not be contiguous, IIRC it was split - so you might find that there's some other stuff in the middle that you will need to avoid corrupting....

[nopeg]

Hint: Found this tool, which was able to replace the '133' file in the exe. http://www.angusj.com/resourcehacker/

Yes, I used this tool alongside HxD (https://mh-nexus.de/en/hxd/) to manually modify the pattern tables (in those headers in 129 file)

Q2. Does the pedal freak out when there are less (or more) items in the 'table' (with names/samples)?

It seems to be working fine when I change any item:

1. this is a GUIDE pattern, which i renamed to 'TEST' and changed kick sample to snare and pedal output as expexted (the name bar changed to 'TEST' and every kick sound changed to snare)

2. this time i erased the snare sample and the sound didn't play as expected

3. this time i just pasted kick in random locations and none of them played as expected

4. this time i changed the pointer to 8Beats1 and it played the 8beat pattern with slightly modified samples

I have a nagging memory that the data for the drums may not be contiguous, IIRC it was split - so you might find that there's some other stuff in the middle that you will need to avoid corrupting....

yeah thats a problem. is there any way to tell which of the data is drum bytes?

[mungewell]

Looks like you are making good progress. I am travelling this week so away from my equipment.

I didn't understand what you were trying to achieve with '3'. Just adding the 'Kik1.raw' name would not cause the sound, and it looks like the file name was added in the wrong place/address. The space for each of these names are 12 bytes long.

Triggers would have to be added to the drum Pattern (ie where the pointer points to) - which is a list of Samples (wrt to filenames above), each declaring which sample to play, at what volume and how long before the next.... I'm not sure/don't remember what units skiptime is in.

https://github.com/mungewell/zoom-zt2/blob/master/decode_patterns.py#L37

I'll refer back to the ASCII are representation; if you modify the Pattern you should be able to see anything you add/delete.

[nomadbyte]

...The actual value of the pointer is uncertain.

Indeed it's not clear how the patterns data is being assembled and referenced after it's loaded, though this somewhat resembles section addressing (or resource references).

If you look closer at the file offset 0x63700, there is stored the section's base offset 0xc016f780, followed by the section's size: 0x000123a4. This section starts with "Dummy" rhythm descriptor. It ends with "Dummy" rhythm's sequence data. In general the section includes other binary data along with the rhythm patterns.

So, for each rhythm the descriptor includes a section offset ("pointer") of its pattern sequence. To translate this into file offset the following formula can be used:

seq_fileOffset  = base_fileOffset + 0x08 + (seq_sectionOffset - base_sectionOffset)

For G1 FOUR:
seq_fileOffset  = 0x63700 + 0x08 + (seq_sectionOffset  - 0xc016f780)

E.g. ("8Beats1"):
seq_fileOffset  = 0x63700 + 0x08 + (0xc017e140  - 0xc016f780) = 0x720c8

0x08 is to account for the section's base details (2 ints for offset and size).

As for the pattern sequence, if one follows the formula above, it appears that the pattern is a sequence of two-byte events each consisting of the following details:

Event {
  spacing: byte,   // in ticks, 1 beat = 12 ticks
  drumId: MSb4,    // drumId: 1..9 (as defined in rhythm's descriptor), 0xF=end
  level: LSb4
}

This yields somewhat differing file offsets than what's reported from the @mungewell script, but the pattern data is essentially the same; the difference is in the first spacing byte which is always 0x00, at least for G1 FOUR patterns.

Pattern sequence data appears to be compiled in, so replacing it may be limited by length of the available "slots". Spilling over the slot most likely would corrupt the firmware. While most patterns (except for "Metro" and the actual "Dummy" sequence) are two-bar length, yet the number of events differs based on the complexity of the pattern.

It may be fun to try to script some "upload" for rhythm patterns, but it may be too much of an effort for the purpose, as there are plenty of other choices to produce a rhythm track for practice.

By the way, A1 FOUR packs quite a bit differing rhythms and sequences, though lacking fw updates it's not very likely to dig those from the pedal's internals.

P.S. Impressive work effort by @mungewell to dig this out and nicely visualize the rhythm sequence!!