Interface design for pianoroll representation

[lzqlzzq]

We are designing interface for pianoroll representation. There are different existing interface designs:

• miditoolkit :
  • Only accept List[Note] as input. (I think that' s weird because miditoolkit itself has containers as Instrument.
  • resample_factor, resample_method, time_portion, keep_note_with_zero_duration are provided as arguments, because division of it is tpq. (I think we accept Score<Tick>, Track<Tick>, Note<Tick> should be enough)
  • velocity_threshold is provided as argument. (I think we can ignore it and implement filter_notes)
• pypianoroll:
  • Accept encode_velocity as argument to decide whether to binarize the pianoroll.
  • No much arguments.
• muspy:
  • Basically the same as pypinaoroll

I think the interface must satisfy:

• Provide encode_velocity as argument to decide whether to binarize the pianoroll.
• Let user use the global resample interface instead of resolution -related argument.
• Provide pitch_range, pitch_offset to clip pitch range.
• modes of pianoroll (onset, offset, frame), accept as a List[str]?

Here are my considerations, do you have ideas? @Natooz @Yikai-Liao

[lzqlzzq]

And, how to handle overlapped notes?

[Natooz]

I think these are good directions! I agree that it is better to leave to the user the task to use the "MIDI-level" resampling methods.

filter_notes would be a function given by the user?

For overlapping notes, we can either:

• not bothering with them, that would in practice merge them into one;
• insert a "pause" tick between them to mark the separation (trimming the offset/duration of the first note).

Ideally, this could be given as an option to the user.

Also pypianoroll (and muspy?) has nice visualisation methods. Do you think it could be worth adding similar features in symusic?

[Yikai-Liao]

filter_notes will be added, which accepts a python function, like a lambda funciton. But just like using python lambda funciton in sort, don't expect it to be as fast as native c++ code.

For the pause token, do you mean this kind of things :

Note(start=0, duration=2, pitch=60) Note(start=2, duration=2, pitch=60)

• without pause [1, 1, 1, 1]
• with pause [1, 0, 1, 1]

Visualization part would be hard in cpp part. And I think it would better to reuse the code in pypianoroll (using matplotlib) to create such image. To be honest, we don't need to take care of the performance of visualization part.

Further, I think it would be better to add a install option for visualization like pip install symusic[visualizaiton] and pip install symusic[all]. Because many people just never use this part (visualization) and matloplib is large.

[Natooz]

You got it right for the pause ticks.

I agree with you for everything for the visualisation. It's indeed a part used only by a small proportion of users. (I was just asking out of curiosity, not recommending or asking for anything, thought you might be interested)

For filter_notes, maybe I'm missing something, but if the piano roll method receives a Score or Track object and this function, why not just letting the user filter the Score/Track himself with his own method before calling the piano roll method?

[Yikai-Liao]

filter_notes is for Score and Track (not for piano roll), it is implemented in some older versions and removed for some reasons that I have forgotten. It will be reimplemented in future versions.

[lzqlzzq]

I am implementing the pianoroll feature. Because the de-overlap operation is a little complicated and seems to be a useful tool even without the pianoroll, I plan to put it in ops.h. And, how to handle the following situation if a "pause" tick must be added?

Note(start=0, duration=1, pitch=60) Note(start=1, duration=2, pitch=60)

The first note cannot be ignored, and shifting the start of a note seems to change the semantics.

[lzqlzzq]

The only way I think is to raise a warning.

[Yikai-Liao]

I am implementing the pianoroll feature. Because the de-overlap operation is a little complicated and seems to be a useful tool even without the pianoroll, I plan to put it in ops.h. And, how to handle the following situation if a "pause" tick must be added?

Note(start=0, duration=1, pitch=60) Note(start=1, duration=2, pitch=60)

The first note cannot be ignored, and shifting the start of a note seems to change the semantics.

In my opinion, pause could only be used when ticks_per_quarter is large, which means the situation where duration=1 would be rare, and reduce the duration by 1 tick won't change the semantics too much.

When we need to use a "strong quantization" for creating pianoroll, the best way to deal with this kind of situation should be using onset pianoroll with frame pianoroll or offset pianoroll at the same time. For example, just put two kind of pianoroll in two distinct channels (this refers to the concept of channel in the images), and input them together into a model (e.g. CNN)

BTW, the example is actually not de-overlap, it is de-adjacentize (or de-adj for short)

This is the situation for de-overlap: Note(start=0, duration=2, pitch=60) Note(start=2, duration=2, pitch=60)