Can't sync midi file notes with audio file

[DanieleCali]

Hi, first of all, good job for this library and the support you always give to the users.

I'm new to unity and I'm having fun trying to make a guitar hero style game where there are notes spawned in lines and you have to press the correct button at the correct moment. To do it, I'm using midi files to have the map of the notes and an audio file to play the real song above the notes. The audio file is a simple mp3 file so it doesn't contains any information of the notes.

Following the issue #79 I managed to spawn and visualize in sync notes in my game, but when I tried to add the real audio file to the game, I notice that after a few seconds, the visualized notes and the audio get out of sync and I can't understand why.

With my little knowledge, I would expect that the midi file notes would map perfectly the audio file notes, but this isn't the case... So I tried to fix this issue by matching the midi file position to the audio source file position but I still can't get my game in sync.

This is the piece of code I wrote:

    # To delay the start of the music by X seconds
    [SerializeField] private float startAudioSourceDelayInSeconds = 4;

    # To delay the start of the playback by X seconds (it can be different from the audio delay)
    [SerializeField] private float startPlaybackDelayInSeconds = 5;

    private float dspSongTime;

    # I'm basing the sync with the unity audio system to be more precise
    void Awake() {
        dspSongTime = (float) AudioSettings.dspTime + startAudioSourceDelayInSeconds;
        audioSource.PlayScheduled(dspSongTime);
    }

    # I'm using to "MoveToTime" function to move the midi file to the audio source position each frame
    private void Update() {
        long positionInMicroseconds = (long)((AudioSettings.dspTime - dspSongTime + startPlaybackDelayInSeconds) * 1000000); // Convert seconds to microseconds
        var positionInTicks = new MetricTimeSpan(positionInMicroseconds);
        _playback.MoveToTime(positionInTicks);
    }

I would also share how I'm transposing the notes from top to bottom:

    # The initialPositionY is where the note is spawned
    private void Update() {
        float yPosition = initialPositionY - _playback.GetCurrentTime<MetricTimeSpan>().TotalMicroseconds / 100000.0f;
        transform.position = new Vector3(transform.position.x, yPosition, transform.position.z);
    }

Have you any idea on how I could reach my goal?

[melanchall]

Hi,

It's hard to say where a problem is since I'm not good at Unity. Can you please record a video of what's going on to make the issue more clear?

Thanks, Max

[DanieleCali]

Video Sample

Here it is.

You can see that it starts in sync with the song but after a couple of seconds it becomes always more out of sync.

[melanchall]

Thanks!

Here some suggestions:

1. Can you put two text labels on the game screen where first one shows current time of an audio file and second one - current time of a MIDI file?
2. Are you sure audio file and MIDI one are in sync? I mean without starting the game. Put the audio file and the MIDI one on two tracks in DAW, for example, and hit the Play button.

[DanieleCali]

1. I don't know if you can see the logs in the video, It prints the difference between the playback position and the audio source position in seconds, the code is:

   var difference = System.Math.Round((_playback.GetCurrentTime<MetricTimeSpan>().TotalMicroseconds / 1000000) - ((AudioSettings.dspTime - dspSongTime + startPlaybackDelayInSeconds)), 5);
   Debug.Log("Difference between playback time and audio source time: " + difference);

   The difference is always 0.XXX, it doesn't increase.

2. I tried to manually start the audio and the midi file simultaneously and I think they are not in sync. But how can I find 2 files completely in sync? maybe the issue is just this... And how the people can make rhythm games where the notes and the audio are completely in sync if using midi files?

[melanchall]

But how can I find 2 files completely in sync? maybe the issue is just this... And how the people can make rhythm games where the notes and the audio are completely in sync if using midi files?

I see following solutions:

1. Create an audio file from a known MIDI file.
2. Adjust the tempo of a MIDI file to make sure it's in sync with an audio file. I think it's not so difficult in DAW.
3. Ask these questions to people from similar projects. For example, Clone Hero has large community in Discord. You can ask them how they prepare files.

Also this code looks not good in Update method:

_playback.MoveToTime(positionInTicks);

Update is called every frame. You probably think that MoveToTime is a cheap operation but it's not. This method stops playback, rewinds playback's pointer to required event and starts the playback again. Doing these operations every frame looks wrong. It's not a source of your issue, just for your information.

By the way, can you please provide the audio file and the MIDI one you use for testing?

Thanks, Max

[DanieleCali]

Thanks for the suggestions! I'll try to follow them even if, right know, I have no idea how to do them 😄

Thanks also for the information on MoveToTime function, I already suspected that this function shouldn't be called each frame but it was the only idea that came in my mind 😄

Here there is the .zip containing the midi and the mp3 file.

Sweet Child Of Mine.zip

[melanchall]

Thanks for the archive. I'll check it out next week.

[melanchall]

Yes, the files are not synced. First of all, align their starts:

And then just find relaxing part so waveform have noticeable peaks:

We will see at part that starts at 4:40 in the mp3 file after aligning with the MIDI one. This part starts in the middle of bar 150 in audio file. But the same part starts in MIDI file in the middle of bar 147. So the MIDI file is faster than the audio one.

[DanieleCali]

So, you suggest to work on a DAW to sync midi and song file ok. I just have to study how to do it 😄 Thanks again with the support!

[melanchall]

So can the issue be closed?

[DanieleCali]

yes, thank you.

[DanieleCali]

Just to update this question. I've downloaded waveform free as DAW and manually changed the tempo of both midi and audio file for each bar in order to get them in sync. Now it is also working on my game. Thanks again for the support, have a good day 😄