Visualization of notes

[ggenije]

I am trying to recreate something like this: https://www.youtube.com/watch?v=F9tHW67f6T0 It's visualization of some midi file, so I need positions of all notes in particular time. Any tips? Also I was unable to pause playback. And how to even synchronize objects and the music.

[melanchall]

Hi,

Thanks for using the library. Let's clarify what problems do you have.

1. Do you need to get all notes at specific time? What units of time? Hours, minutes, seconds?
2. What do you mean "I was unable to pause playback"? How do you start playback and what you do to pause it?
3. There were issues similar to your: #3 and #5. Please read discussions on those issues. Maybe you'll find ideas regarding how to sync audio and midi.

[ggenije]

This is my test code and playback is working (I'm working in unity) First problem: It seems that song isn't played as expected, some notes doesn't stop just as simply as said.

using Melanchall.DryWetMidi.Core;
using Melanchall.DryWetMidi.Devices;
using Melanchall.DryWetMidi.Interaction;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEditorInternal;
using UnityEngine;

public class TestScript : MonoBehaviour
{

    public GameObject gameNoteObj;

    private Playback _playback;
    private OutputDevice _outputDevice;

    // Start is called before the first frame update
    void Start()
    {
        var midiFile = MidiFile.Read("Assets/HangEight_SylvielmnaZjsch.mid");
        _outputDevice = OutputDevice.GetById(0);
        _playback = midiFile.GetPlayback(_outputDevice, new MidiClockSettings
        {
            CreateTickGeneratorCallback = interval => new RegularPrecisionTickGenerator(interval)
        });

        _playback.NotesPlaybackFinished += Test;
        _playback.InterruptNotesOnStop = true;
        StartCoroutine(StartMusic());

    }

    private void Test(object sender, NotesEventArgs notesArgs)
    {
        var notesList = notesArgs.Notes;
        foreach (Note item in notesList)
        {
            Debug.Log(item);
        }
    }

    private IEnumerator StartMusic()
    {
        _playback.Start();
        while (_playback.IsRunning)
        {

            yield return null;

        }
        _playback.Dispose();

    }

    private void OnApplicationQuit()
    {
        _playback.Stop();
        _playback.Dispose();
    }
}

Second problem: But I'm unable to start the playback from somewhere else rather than start ( which is same problem as pausing basically). didn't tried anything because I don't know what I am doing.

And the basically I need want to do is to visualize notes in way so that every note is spawned at beginning of playing( in form of GameObjects - the thing user sees as rectangles which size depends on length of the note, x position is time of that note in playback, and the y position is the frequency of the note). Then all notes moves to the left, (actually camera is moving to the right) so that way when the left side of the notes passes certain position (determined by x position, which can be left edge of the screen for example) it starts playing , and the users gets another visual indication that that notes is playing current time in sight of some particle effect for example.

[melanchall]

Thanks for clarification. I have some more questions.

some notes doesn't stop

As I see from your code, the only place where you stop playback is OnApplicationQuit. If you quit app, some notes still playing?

I'm unable to start the playback from somewhere else

From where, for example? And what happens if you start it from that place?

I need want to do is to visualize notes in way so that every note is spawned at beginning of playing

Playback has NotesPlaybackStarted and NotesPlaybackFinished events. You can subscribe to them and draw rectangles and other visual things when notes started or finished playing.

Also please provide MIDI file you test your code on, and say what Unity version you use.

[ggenije]

They stop after application quit. I am unable because I don't how. I must draw notes that will need to come so the user can see even before they play/.

Unity 2019.3.0f3 with API Compatibility level .NET 4.x Link of midi: https://www.vgmusic.com/file/0a9811c676b79a8911b39526a7e3dcb6.html

[ggenije]

Also sometimes this error occurs: IndexOutOfRangeException: index + length > size System.Array.Clear (System.Array array, System.Int32 index, System.Int32 length) (at <437ba245d8404784b9fbab9b439ac908>:0) System.Collections.Generic.HashSet`1[T].Clear () (at <351e49e2a5bf4fd6beabb458ce2255f3>:0) Melanchall.DryWetMidi.Devices.Playback.Stop () (at Assets/DryWetMidi/Devices/Playback/Playback.cs:440) TestScript.OnApplicationQuit () (at Assets/TestScript.cs:59)

[ggenije]

Here is how it sounds like: https://drive.google.com/file/d/1zvzPFOv4pRIHgmC0_zM7OS-KmGxzEgd1/view?usp=sharing (In worst case).

[melanchall]

Thank you for all information. I'll investigate problems and let you know.

[melanchall]

Regarding IndexOutOfRangeException exception: I've made some changes in Playback class to handle multi-threaded access. Please download the latest code from develop branch and check whether error is gone or not.

---

Regarding

It seems that song isn't played as expected

please try:

1. Call _playback.TickClock(); before yield return null;
2. If it's not helped, let's try custom tick generator. Complete code:

using System;
using System.Collections;
using System.Threading;
using Melanchall.DryWetMidi.Core;
using Melanchall.DryWetMidi.Devices;
using Melanchall.DryWetMidi.Interaction;
using UnityEngine;

public class TestScript : MonoBehaviour
{
    private sealed class ThreadTickGenerator : ITickGenerator
    {
        public event EventHandler TickGenerated;

        private Thread _thread;

        public void TryStart()
        {
            if (_thread != null)
                return;

            _thread = new Thread(() =>
            {
                var stopwatch = new System.Diagnostics.Stopwatch();
                var lastMs = 0L;

                stopwatch.Start();

                while (true)
                {
                    var elapsedMs = stopwatch.ElapsedMilliseconds;
                    if (elapsedMs - lastMs >= 1)
                    {
                        TickGenerated?.Invoke(this, EventArgs.Empty);
                        lastMs = elapsedMs;
                    }
                }
            });

            _thread.Start();
        }

        public void Dispose()
        {
        }
    }

    public GameObject gameNoteObj;

    private Playback _playback;
    private OutputDevice _outputDevice;

    // Start is called before the first frame update
    void Start()
    {
        var midiFile = MidiFile.Read("Assets/HangEight_SylvielmnaZjsch.mid");
        _outputDevice = OutputDevice.GetById(0);
        _playback = midiFile.GetPlayback(_outputDevice, new MidiClockSettings
        {
            CreateTickGeneratorCallback = _ => new ThreadTickGenerator()
        });

        _playback.NotesPlaybackFinished += Test;
        _playback.InterruptNotesOnStop = true;
        StartCoroutine(StartMusic());
    }

    private void Test(object sender, NotesEventArgs notesArgs)
    {
        var notesList = notesArgs.Notes;
        foreach (Note item in notesList)
        {
            Debug.Log(item);
        }
    }

    private IEnumerator StartMusic()
    {
        _playback.Start();
        while (_playback.IsRunning)
        {
            yield return null;

        }
        _playback.Dispose();
    }

    private void OnApplicationQuit()
    {
        _playback.Stop();
        _playback.Dispose();
    }
}

Please tell me how MIDI file is played with these changes.

---

I'm unable to start the playback from somewhere else I am unable because I don't how.

I'm not expert in Unity so I suppose you should read Unity documentation to learn what game methods are available for you by Unity, and decide where you want to start playback. To start playback from desired place, move this line

_playback.Start();

from StartMusic to that desired place.

---

I must draw notes that will need to come so the user can see even before they play

Just draw notes, start game (start moving camera) and then start actual playback after X seconds. X is a number of seconds the first rectangle should reach "play line" after.

[ggenije]

It's working now (With custom ticking)! I don't hear any errors in the MIDI. I'll try to realize now what I wanted. The only error is Disposing playback after MIDI ends. "ObjectDisposedException: Cannot access a disposed object.", but that is not problem of DryWetMidi but it's about implementation of the my current script.

[ggenije]

Edit: I managed to do it: using _playback.GetCurrentTime MidiTimeSpan().TimeSpan

_Also I'm not having trouble with Unity... The real problem is how to convert note time system to space: How can I get of the current playback in same unit as note.Time ? I am trying playback.GetCurrentTime but i can't get actually value

[ggenije]

Timing issue: https://drive.google.com/open?id=12bQ5ug85if-eV-sof-nCGP3e8I-QZweJ Is it how MIDI works or there is problem here? You can see that _playback.GetCurrentTime< MidiTimeSpan>().TimeSpan is not increased evenly, so somewhere short notes are played long and vice versa. Music and timing is OK but it looks like it's speedup from time to time. liz_rhap02.zip

[melanchall]

Looks like camera's moving isn't smoothy, sometimes camera jumps back or forward by small amount of distance. How you move the camera? Is it related with playback's current time?

Also please check whether the same behavior exist in game itself (not in Unity editor) or not? Can you build the game and look at the camera moving without Unity? Maybe these jumps are introduced by Unity editor.

If nothing changed, can you give me your Unity project? If you don't want to show it to the whole world here, you can email me.

[ggenije]

Note Object

  public void InitGameNote(float timeOfNote,int noteNumber,float duration,float instrument)
    {
        transform.position = new Vector3(timeOfNote, -noteNumber);
        GetComponent<SpriteRenderer>().size = new Vector2(duration, 1f);
        GetComponent<SpriteRenderer>().color = Color.HSVToRGB(instrument / 10f, 1f, 1f);
    }

Notes positions:

 foreach (var note in Notes)
        {
            GameObject nObj = Instantiate(gameNoteObj);
            nObj.GetComponent<GameNote>().InitGameNote(note.Time/1000f*NoteWidth, note.NoteNumber, note.Length / 1000f * NoteWidth, note.Channel);

        }
//   /1000f * NoteWidth is some constant

Camera Position.

  long previousTime = 0;
    private IEnumerator StartMusic()
    {
        _playback.Start();
        while (_playback.IsRunning)
        {
            var currentTime = _playback.GetCurrentTime<MidiTimeSpan>();

            height = Camera.main.orthographicSize * 2.0f;
            width = height * Screen.width / Screen.height;

            Debug.Log((currentTime.TimeSpan - previousTime) / Time.deltaTime);
            GameCamera.transform.position = new Vector3(currentTime.TimeSpan / 1000f * NoteWidth + width/2f, GameCamera.transform.position.y,GameCamera.transform.position.z);
            previousTime = currentTime.TimeSpan;
            yield return null;

        }
        _playback.Dispose();
    }

The value of (currentTime.TimeSpan - previousTime) / Time.deltaTime; should be constant but it's going from 150 to 750 which is HUGE difference. Also you can see that it's that MIDI and camera position is synchronized meaning camera is always in good position. Also you can see for example: in second 34 in the previous video that note which duration is much longer than others have very small length, concluding that ratio isn't accurate. Some notes which are short sound have long length and vice versa, so it isn't problem with Unity. Maybe it's how MIDI works, but it is not Unity problem for sure.

[melanchall]

If you get different delta between current time and previous, it's obviously related with Unity. Your coroutine StartMusic yields every frame. Frame rate is completely managed by Unity.

To check it, let's measure delta between frames:

long _previousMs = 0;
Stopwatch _stopwatch = new Stopwatch();

private IEnumerator StartMusic()
{
    _stopwatch.Start();
    while (true)
    {
        var ms = _stopwatch.ElapsedMilliseconds;
        Debug.Log(ms - _previousMs);
        _previousMs = ms;
        yield return null;
    }
}

Did you test your game in release build and not in Unity editror, as I've asked?

[ggenije]

It's the same in build as in editor. And about:

long _previousMs = 0;
Stopwatch _stopwatch = new Stopwatch();

private IEnumerator StartMusic()
{
    _stopwatch.Start();
    while (true)
    {
        var ms = _stopwatch.ElapsedMilliseconds;
        Debug.Log(ms - _previousMs);
        _previousMs = ms;
        yield return null;
    }
}

min value is 42ms and max 55ms . Similar results are when I put MIDI playing at that time 44 at average. It's 44ms even in part at second 34 in previous video. There is no connection between those.

[melanchall]

44ms is pretty high value.

Let's try to start playback right in Start method and comment this line:

StartCoroutine(StartMusic());

It's just for test. Does MIDI playback work without coroutine?

Also please remove subscribing to playback's events, like this line:

_playback.NotesPlaybackFinished += Test;

[ggenije]

I just changed Debugging by Debug.Log by UI text it's now 20ms on average in editor and 3.5ms in build mode. Also I already had tested does MIDI works without coroutine and it's the same result.

[melanchall]

Can you give me entire project?

[ggenije]

Can you give me entire project?

Sent to m.hall.muz@gmail.com.

[melanchall]

Thanks. I'll look into it as soon as possible.

[melanchall]

Finally I've looked into your code.

First of all, there are no issues with MIDI file playing, all notes played exactly when they must be played. I've dumped all deltas to file:

timings.txt

where each string is:

F(inished)/S(tarted) ::
note ::
exact start (for S) or end time (for F) of a note in ms ::
playback's current time in ms ::
delta between current timeand exact time

As you can see deltas are 0 or 1 so notes played correctly.

The problem with your game is camera moving. You use MidiTimeSpan for time, length and camera position changing. And that's wrong.

MidiTimeSpan represents time/length in terms of MIDI ticks (please read Standard MIDI File spec on midi.org). BUT each tick can last different amount of milliseconds. It depends on Set Tempo events within a MIDI file. Your file contains a lot of tempo changes (2829), so you get those accelerations and decelerations of camera moving.

You obviously need to rely on metric time (milliseconds). Let's see what changes you need to do.

First of all, initialization of notes should be:

var Notes = midiFile.GetNotes();
var tempoMap = midiFile.GetTempoMap();

foreach (var note in Notes)
{
    GameObject nObj = Instantiate(gameNoteObj);
    nObj.GetComponent<GameNote>().InitGameNote(note.TimeAs<MetricTimeSpan>(tempoMap).TotalMicroseconds / 100000.0f, note.NoteNumber, note.LengthAs<MetricTimeSpan>(tempoMap).TotalMicroseconds / 100000f * NoteWidth, note.Channel);
}

Then, camera changing should be:

while (_playback.IsRunning)
{
    var currentTime = _playback.GetCurrentTime<MetricTimeSpan>().TotalMicroseconds / 100000.0f;

    GameCamera.transform.position = new Vector3(currentTime + width/2f, GameCamera.transform.position.y,GameCamera.transform.position.z);

    yield return null;

}

With these changes camera moves smoothly. Speed of camera moving is controlled by that 100000.0f constant. Change it to achieve desired camera moving speed.

[ggenije]

Great! It's working now. But what about:

 private sealed class ThreadTickGenerator : ITickGenerator
    {
        public event EventHandler TickGenerated;

        private Thread _thread;

        public void TryStart()
        {
            if (_thread != null)
                return;

            _thread = new Thread(() =>
            {
                var stopwatch = new System.Diagnostics.Stopwatch();
                var lastMs = 0L;

                stopwatch.Start();

                while (true)
                {
                    var elapsedMs = stopwatch.ElapsedMilliseconds;
                    if (elapsedMs - lastMs >= 1)
                    {
                        TickGenerated?.Invoke(this, EventArgs.Empty);
                        lastMs = elapsedMs;
                    }
                }
            });

            _thread.Start();
        }

        public void Dispose()
        {
        }
    }

It's working properly with

_playback = midiFile.GetPlayback(_outputDevice, new MidiClockSettings
        {
            CreateTickGeneratorCallback = _ => new ThreadTickGenerator()
        });

but there is problem with

 _playback = midiFile.GetPlayback(_outputDevice, new MidiClockSettings
        {
            CreateTickGeneratorCallback = interval => new RegularPrecisionTickGenerator(interval)
        });

Use this MIDI to test https://www.vgmusic.com/file/0a9811c676b79a8911b39526a7e3dcb6.html

[melanchall]

Every playback (audio, viddeo, MIDI etc) uses timer under the hood. In DryWetMIDI this timer called tick generator. On every tick of timer playback looks what objects should be played by the current time, plays them and advances position within objects list waiting for next tick.

To make playback smooth and correct, precision of timer should be ~1ms. So tick will be generated every one millisecond. By default, DryWetMIDI uses HighPrecisionTickGenerator which is the best option in terms of CPU usage, memory usage and precision (1 ms). But unfortunately it cause problems on Mono (CLR implementation of Unity). Details here: #31.

So we need to provide ticking engine that will work with Unity. RegularPrecisionTickGenerator uses standard Timer class and works with Mono, but its precision is about 16ms on Windows which is bad. So events will be played by grid with step of 16ms. It's known problem of Timer.

ThreadTickGenerator gets you 1ms precision but while (true) loop is not good for CPU. I'll try to provide you another implementation. I have some idea how we can construct it without high CPU usage.

Also I'll do some changes in tick generators API and let you know.

[melanchall]

Unfortunately right now I can't suggest tick generator for Unity that will be workable and better than ThreadTickGenerator. I've made some changes in tick generators API. With latest version of the library code implementation of ThreadTickGenerator is:

private sealed class ThreadTickGenerator : TickGenerator
{
    private Thread _thread;
    private bool _isRunning;
    private bool _disposed;

    protected override void Start(TimeSpan interval)
    {
        if (_thread != null)
            return;

        _thread = new Thread(() =>
        {
            var stopwatch = new Stopwatch();
            var lastMs = 0L;

            stopwatch.Start();
            _isRunning = true;

            while (_isRunning)
            {
                var elapsedMs = stopwatch.ElapsedMilliseconds;
                if (elapsedMs - lastMs >= interval.TotalMilliseconds)
                {
                    GenerateTick();
                    lastMs = elapsedMs;
                }
            }
        });

        _thread.Start();
    }

    protected override void Dispose(bool disposing)
    {
        if (_disposed)
            return;

        if (disposing)
        {
            _isRunning = false;
        }

        _disposed = true;
    }
}

And:

CreateTickGeneratorCallback = () => new ThreadTickGenerator()

Do you have any questions?

[ggenije]

So now I just have to add new version and use ThreadTickGenerator instead of RegularPrecisionTickGenerator(And to remove the ThreadTickGenerator from my script)?

[melanchall]

I suppose you already use ThreadTickGenerator. Just take latest code of the library, change implementation with that I've provided above and use

CreateTickGeneratorCallback = () => new ThreadTickGenerator()

instead of

CreateTickGeneratorCallback = _ => new ThreadTickGenerator()

[ggenije]

After taking last code from library from 4/15/2020 and after deleting private sealed class ThreadTickGenerator : ITickGenerator and renaming as you said before; There is error which says that ThreadTickGenerator doesn't exists?

[melanchall]

I didn't say that you need rename it, just change implementation.

OK, send me new TestScript file and I'll fix it for you.

[ggenije]

renaming _ into () that I meant

using System;
using System.Collections;
using System.Threading;
using Melanchall.DryWetMidi.Core;
using Melanchall.DryWetMidi.Devices;
using Melanchall.DryWetMidi.Interaction;
using UnityEngine;
using UnityEngine.UI;
using UnityEngine.UIElements;

public class TestScript : MonoBehaviour
{

    public GameObject gameNoteObj;

    private Playback _playback;
    private OutputDevice _outputDevice;

    public GameObject GameCamera;

    float height;
    float width;

    const float NoteWidth=0.5f;

    public Text testText;

    // Start is called before the first frame update
    void Start()
    {
        var midiFile = MidiFile.Read("Assets/liz_rhap02.mid");

        _outputDevice = OutputDevice.GetById(0);

        _playback = midiFile.GetPlayback(_outputDevice, new MidiClockSettings
        {
            CreateTickGeneratorCallback = () => new ThreadTickGenerator()
        });

        var Notes = midiFile.GetNotes();
        var tempoMap = midiFile.GetTempoMap();

        foreach (var note in Notes)
        {
            GameObject nObj = Instantiate(gameNoteObj);
            nObj.GetComponent<GameNote>().InitGameNote(note.TimeAs<MetricTimeSpan>(tempoMap).TotalMicroseconds / 100000.0f, note.NoteNumber, note.LengthAs<MetricTimeSpan>(tempoMap).TotalMicroseconds / 100000f * NoteWidth, note.Channel);
        }

        // _playback.NotesPlaybackFinished += Test;
        _playback.InterruptNotesOnStop = true;

        StartCoroutine(StartMusic());
    }

    private void Test(object sender, NotesEventArgs notesArgs)
    {
        var notesList = notesArgs.Notes;
        foreach (var item in notesList)
        {

        }
    }

    long _previousMs = 0;
    float average = 0;
    float avCount = 0;
    System.Diagnostics.Stopwatch _stopwatch = new System.Diagnostics.Stopwatch();
    private IEnumerator StartMusic()
    {
        _playback.Start();
        _stopwatch.Start();
        while (_playback.IsRunning)
        {
            var currentTime = _playback.GetCurrentTime<MetricTimeSpan>().TotalMicroseconds / 100000.0f;

            height = Camera.main.orthographicSize * 2.0f;
            width = height * Screen.width / Screen.height;
            var ms = _stopwatch.ElapsedMilliseconds;
            average += ms - _previousMs;
            avCount++;
            testText.text = (ms - _previousMs).ToString()+"\n"+(average/avCount).ToString();
            GameCamera.transform.position = new Vector3(currentTime + width / 2f, GameCamera.transform.position.y, GameCamera.transform.position.z);

            _previousMs = ms;
            yield return null;

        }

        _playback.Dispose();
    }

    private void Update()
    {
        if (Input.GetKey(KeyCode.DownArrow))
            GameCamera.transform.position += new Vector3(0f, -Time.deltaTime*10f, 0f);
        else if (Input.GetKey(KeyCode.UpArrow))
            GameCamera.transform.position += new Vector3(0f, Time.deltaTime*10f, 0f);

        if(_playback.IsRunning)
        {

            if (Input.GetKeyDown(KeyCode.LeftArrow))
            {
                if(_playback.GetCurrentTime<MetricTimeSpan>().TotalMicroseconds>5000000)
                {
                    _playback.Stop();
                    _playback.Play();
                    _playback.MoveBack(new MetricTimeSpan(5000000));
                }

            }
            if (Input.GetKeyDown(KeyCode.RightArrow))
            {
                if (_playback.GetCurrentTime<MetricTimeSpan>().TotalMicroseconds + 5000000 < _playback.GetDuration<MetricTimeSpan>().TotalMicroseconds)
                {
                    _playback.Stop();
                    _playback.Play();
                    _playback.MoveForward(new MetricTimeSpan(5000000));
                }

            }
        }

    }

    private void OnApplicationQuit()
    {
        if(_playback.IsRunning)
        {
            _playback.Stop();
            _playback.Dispose();
        }

    }
}

[melanchall]

Please read my comment again: https://github.com/melanchall/drywetmidi/issues/79#issuecomment-614728271

I didn't say to remove ThreadTickGenerator. I gave you new implementation that you should use. Why did you remove ThreadTickGenerator code?

[ggenije]

When you said:

I've made some changes in tick generators API.

I thought you said

I implemented ThreadTickGenerator so download new version of DryWetMidi to use it

Sorry, you can close the issue now.

[melanchall]

Thanks for the issue! Feel free to open new ones :)

[jrdndj]

Can you give me entire project?

Sent to m.hall.muz@gmail.com.

hi! Ive been following this issue as well. If it is not too much, may I ask for the code of the complete project as well? Im trying to recreate the same piano roll as well and this would greatly help me.

[melanchall]

@ZdravPrav can I give the sample project to @jrdndj ?

[jrdndj]

If you could maybe send the resources to jrdn.deja@gmail.com so I could check them out. Im a starter at unity so seeing the code would at least help me build the piano roll visualizer :D