search

mtolly / jammittools

Export sheet music (PDF) and audio (WAV) from the Windows/Mac app Jammit
GNU General Public License v3.0
31 stars 5 forks source link

synchronization between the audio and the score #8

Closed aanilpala closed 7 years ago

aanilpala commented 7 years ago

firstly, thanks for the great project. Cool stuff.

I was wondering if it is possible to extract the alignment data between the audio tracks and the measures (or start of the line) of the score from jcf files. I believe this info must be stored somewhere in jcf files so that jammit was able to display a moving cursor on the sheet as the audio plays. I observed that the cursor speed depends on the click track (bpm of the click track must be proportional to the cursor speed which can vary from measure to measure within a track depending on the tempo)

mtolly commented 7 years ago

Hey, sorry I missed this. There are 2 places where those appear to be stored.

First is the beats.plist file, which contains a list of positions in seconds, and then some of them are marked with isDownbeat signifying the first beat of a measure. I believe the sheet music consistently has 2 measures per line, so from that you can figure out how long each sheet music line takes in seconds. I implemented a simple cursor based just on this in a proof of concept Jammit player: https://github.com/mtolly/jelly

The trickier part is where the beats fall within each line, which I believe is stored in nowline.nodes. This is a binary file that almost certainly has some kind of mapping between lines+pixels and timestamps in seconds. But I haven't dug into it to try to decode it for sure.