[Research] Proper BPM calculation for XSQ charts

I'm working on a tool for some of Dance Maniax 2nd's data right now and it involved having to figure out the proper method for calculating BPM and absolute beat position based on the absolute timestamp, among other things.

I looked at Dance Maniax Update first as a quick reference and saw that the XSQ chart reader doesn't seem to handle BPMs properly and uses a hack for BPM changes, so I am writing the information needed to hopefully help fix it properly. I don't have an environment set up currently to build the program so I can't send over a PR so I hope an issue is good enough.

https://github.com/AllenSeitz/dance-maniax-update/blob/749bb19d15d475a7cb01e62dbe97aad15f1d3d03/src/source/xsq_read.cpp#L12-L32

The XSQ_RECORD struct should look like this:

    int timestamp;
    int beat;
    unsigned short next_bpm_entry_index;
    short column;
    int word4; // always 0x0000FFFF
    int word5; // always zero

timestamp: You already have this correct but the variable name should probably be changed to differentiate it from the beats value. The timestamp is related to the absolute real timestamp in millisecond using (timestamp*60)/18. So if the timestamp is for example 21100 then it would come out to (21100*60)/18=70333ms, or 70.333s.

beat: Divide the beat value by 0x600 to get the position of the beat position in the chart. For example, if the beat value is 0x40800 then 0x40800 / 0x600 = beat 172.

next_bpm_entry_index: An index into an array of XSQ_RECORD values. The entry pointed at by this value is used to calculate the absolute beat position in a song for any given timestamp value. It can also be used to calculate the proper BPM for each section of the chart.

Real world example from the "I Will Follow Him" chart:

The first XSQ entry of the first chart is: 9D 00 00 00 00 00 00 00 6D 00 FE FF 00 00 FF FF 00 00 00 00 timestamp = 0x9d (523 ms) beat = 0 next_bpm_entry_index = 0x6d

The XSQ entry at index 0x6d is: 6C 52 00 00 00 08 04 00 6E 00 22 00 00 00 FF FF 00 00 00 00 Timestamp = 0x526c (70333 ms) Beat = 0x40800 (beat 172) next_bpm_entry_index = 0x6e

The XSQ entry at index 0x6e is: 6B 54 00 00 00 20 04 00 79 00 44 00 00 00 FF FF 00 00 00 00 Timestamp = 0x546b (72036 ms) Beat = 0x42000 (beat 176) next_bpm_entry_index = 0x79

etc

From there you can find the exact BPM by calculating using the difference between the beats and timestamps. bpm = ((next_beat - beat) / 0x600) * (60000 / (((next_timestamp - timestamp) * 60) / 18))

Using the above information, all of the BPM changes for "I Will Follow Him" map out as the following (note: ms values rounded for readability):

523.3 ms, beat 0 -> 70333.3 ms, beat 172 = 147.829824 bpm
70333.3 ms, beat 172 -> 72036.7 ms, beat 176 = 140.900196 bpm
72036.7 ms, beat 176 -> 80150 ms, beat 196 = 147.904684 bpm
80150 ms, beat 196 -> 80610 ms, beat 197 = 130.434783 bpm
80610 ms, beat 197 -> 81120 ms, beat 198 = 117.647059 bpm
81120 ms, beat 198 -> 81693.3 ms, beat 199 = 104.651163 bpm
81693.3 ms, beat 199 -> 82400 ms, beat 200 = 84.905660 bpm
82400 ms, beat 200 -> 85580 ms, beat 204 = 75.471698 bpm
85580 ms, beat 204 -> 91986.7 ms, beat 212 = 74.921956 bpm

Extra: The game calculates calculates the exact beat position given a timestamp in a way that I wasn't expecting so it's worth making a mention here. The exact calculation for absolute beat position given a timestamp is:

# Python example
while next_beat_offset - base_beat_offset > 0xfffe:
    base_timestamp = (next_timestamp + base_timestamp) // 2
    base_beat_offset = (next_beat_offset + base_beat_offset) // 2

absolute_beat = base_beat_offset + int(((next_beat_offset - base_beat_offset) * (requested_timestamp - base_timestamp)) / (next_timestamp - base_timestamp))

It averages the base timestamp and beat values until the difference between the base beat value and the next beat value is <= 0xfffe.

You could also calculate it without the averaging and it will be roughly correct, but the averaging matters to get frame accuracy for videos because certain video commands work based on the beat position instead of the timestamp position (I'll cover this more in another issue later). The difference between averaged vs unaveraged calculations grows the larger the difference between the base beat value and next beat value is above 0xfffe.

Oh my gosh! Thank you! You're the hero I needed 10 years ago.

A long time ago I set out to extract and accurately playback the movie and chart data. Surprisingly the movie scripts weren't too hard? I'm not sure if I really did them correct, but Happy Hopper seems accurate, so alright.

A kind person extracted the mp3s and video data for me. Which was a godsend because there was no way I was ever going to figure out how to decode the mp3s.

So that leaves the chart data. I focused my search on the 16MB of flash memory because I knew it had to be in there somewhere. At first I was thinking that the charts might be "in between" the textures, like DDR. But then somehow I figured out, without proof, that all the charts must instead be in a single compressed blob. (Was I right?)

I tried to locate the chart data by mapping out the entire flash memory. Fortunately a good half of it (iirc) is two video files for fast loading at the start of a song. And after masking out the executable and textures I had limited my search space to about a third, which was also fragmented! But despite those huge hints I still couldn't find the charts. It was all just random numbers with no obvious patterns or poorly compressed bits. I knew there had to be a file system but I'm just a hack who can't actually reverse engineer things. The only tool in my toolbox is "just eyeball it".

What I ended up doing was loading the game in MAME and dumping the charts from memory. MAME at the time basically softlocked whenever it needed to load an mp3. So I'd pick a song, wait for the softlock, then procdump MAME in that moment. Then sift through the dump for "hey that looks like chart data" and save it. After getting Mad Blast and Heaven is a 57 correct I was confident enough to write a program to do it. (Well, sort of automated. DieKatze88 once spent a slow day at work manually creating 4*num_songs of procdumps for me. What a legend!)

These dumps worked flawlessly with gap=0 and the official mp3s. (As they should.) Except I knew I was missing the BPM data. I looked for it separately in the memory dumps but I couldn't find it. And so I bodged it. I said no one will ever notice, And I'm so genuinely happy that after all this time I finally got caught! :)

So now that you know how I obtained my xsq files, can I ask you for cleaner ones please? I thought about asking you on Twitter or on tcrf.net. But you're kind of an elusive person and I didn't want to beg.

Development on this project has clearly stalled. I have so much to do now and being overwhelmed with all of it is keeping me from doing any of it. I have to port to 64 bit and support 720p (I don't know how that's going to work, besides pillarboxes). I have to not hardcode the server url and change the update code to use an API that the Internet still supports. And I have to replace my video playback library with something that still works. And only then can I begin making feature changes to the game itself.

But for the day when/if I get around to it, clean chart data would be nice. I'm still using fan-made dwi files for 1st mix and I'm actually missing the licensed song that was cut.

A long time ago I set out to extract and accurately playback the movie and chart data. Surprisingly the movie scripts weren't too hard? I'm not sure if I really did them correct, but Happy Hopper seems accurate, so alright.

My actual goal was the movie scripts (I've been slowly working on writing tools for rhythm game movie scripts the past few years, starting with some old GFDM games, then after that Sys573 MAX+ era DDR, and more recently DMX after some requests). I did take a look at the video script code in Dance Maniax Update before I started and sadly the video scripting code in DMXU doesn't implement a lot of things, but overall the video scripting isn't too complicated. I planned on writing up a separate issue and sending that info over when I finished it + linking the tool for reference once I released it.

I followed you on Twitter. Send me a DM with your Discord and I'll help you get the data situation sorted out. https://twitter.com/_987123879113

The short and public answer for data would be: Most of my tools for the older Konami rhythm games is in this repo: https://github.com/987123879113/gobbletools/

sys573/py573a is used to decrypt MP3s
sys573/sys573tool dumps the GAME.DAT/flash DAT file by parsing the internal file table and decompressing data as needed. The charts for DMX 1st are ".seq" files (and .lmp for the lamp data), and the charts for DMX 2nd don't have an extension and the movie files are ".mbk". For example for Absolute it would be abso.seq for 1st, abso for DMX 2nd (and abso.mbk for the movie).

You can get everything you would need from that I think. The memory dumped files are slightly different from the actual files in the game's data such as charts being one single file with a header to separate them, and the movie files were dumped at the wrong offset so the data is all shifted.

I pushed my code to my repository, it includes more detailed instructions on how to extract the data as required. Steps 1 until 3 will allow you to extract the GAME.DAT's contents, and from there you can grab all of the source files in their uncompressed form.

https://github.com/987123879113/gobbletools/tree/master/sys573/dmxanimtool

I tested a bunch of random videos and they all seemed good when compared to real hardware footage I had access to.

I will try to document it in more detail later this week when I get the chance.

AllenSeitz / dance-maniax-update

[Research] Proper BPM calculation for XSQ charts #6