Highlight playback position for external playback

A bit of background

I'm developing an application which largely mimics Smart Pianist app from Yamaha. It's purpose is to control digital pianos. With the app user selects a song (midi-file) and the app transmits it to piano. Then there are several modi:

listening mode: piano plays back the song on its own;
practice mode: piano plays back backing part and left hand (or right hand) and the user plays the other hand. Piano waits for user to depress correct keys;
user plays both hands in his own tempo.

In all cases the app doesn't control the playback which happens externally to the app. Piano informs the app about current playback position via MIDI events. These events contain Measure number and Beat number. The original Yamaha's app automatically scrolls the score view and highlights current beat like this:

smart pianist-score view

Questions

What possibilities we have in Lomse to scroll the view into certain position measured as Measure/Beat? How to calculate Lomse cursor position from measure and beat?
Is it possible to highlight the position in such a fancy way like Smart Pianist app does?

Thanks.

Scrolling to measure/beat

What possibilities we have in Lomse to scroll the view into certain position measured as Measure/Beat? How to calculate Lomse cursor position from measure and beat?

Lomse was designed to satisfy the needs of my application: rendering documents containing scores, editing them and playing back the scores via MIDI. But Lomse design is modular and I tried to minimize coupling between different objects and parts, so that other scenarios and needs could be implemented without difficulty (I hope).

Current API does not permit to scroll a view into certain position given as Measure/Beat information. This is because I didn't need that and also the view assumes that it view contains not only scores but full documents with texts, lists, images, etc. Therefore, current approach assumes that the document is displayed as one or more sheets of paper, one after the next one, and scroll position is referred by physical coordinates in real world (that is, scroll is just changing the viewport origin).

Implementing scroll based on measure/beat is possible and should not be complex: just a method Interactor::scroll_to measure(int measure, int beat=1). This method will:

be restricted to documents containing only one score, such as MusicXML imported files.
delegate on GraphicView for determining the physical position of the desired measure/beat.
Pass this information to the GraphicView object, to set up the new viewport.

The main coding task is in GraphicView, for coding the method to determine the physical position of the desired measure/beat. For this, the information in GraphicModel object must be used. This method should be similar to current method GraphicView::change_viewport_if_necessary(ImoId id) in which the GraphicModel is used to determine the position of the note/rest whose Id is passed. Information about measure should not be complex to obtain but information about beats could require more coding, as the GraphicModel knows nothing about beats. The concept nearest to 'beats' is the table TimeGridTable, maintained in every system, relating times with positions. If the time signature is known, converting measure/beat to time could be simple and then the TimeGridTable will provide position.

But the concept of 'beat' is tricky, as it could depend on how the music is conducted: you would like, for instance, to subdivide the pulse for studying better, or to count using half notes instead of quarter notes. There is also the problem of music without time signature (i.e. some very commonly studied Erick Satie piano pieces). Perhaps, before starting playback the app should inform lomse about what is a beat (i.e a quarter note, a half note, a dotted quarter note, etc.).

In any case, I need to study the solution more in depth, but I do not foresee great problems for coding this and without much work, apart from finding a clean and simple solution for capturing the notion of beat.

Highlight the beats

Is it possible to highlight the position in such a fancy way like Smart Pianist app does?

In Lomse, highlight can be also customized. At present, lomse includes two highlight effects. The default option is to change the colour of the noteheads as they are played back. But it is also possible to draw a vertical line that advances to the note being played back. Drawing a rectangle instead of a line is practically the same code and should not require much effort. And positioning the line/rectangle only on beats instead of on each position containing notes is just advancing the highlight with the beats. So I also do not foresee great problems for coding this and without much work.

A third issue: managing all this

Currently, highlight events and events for scrolling the view as playback advances, are generated by the ScorePlayer object at the same time than MIDI events are generated. If Lomse is not going to do the playback then your application must take care of generating and controlling all this by invoking the appropriate Interactor methods when advancing to a new beat.

Conclusion

I do not foresee great problems for coding all this, and should not require much work. My only problem is time. I developed Lomse in my free time when I had a lot of free time, but currently I can only spend time on this mainly on weekends. If you are interested to continue with this and not in a hurry, I could start with the issue of scrolling to measure/beat. Probably in a month could be finished, but I can not assume any commitment for a target date. I will do only unit tests, but if enough time I will code an example to load an score and invoke the scroll method every second to check that it works properly.

Thank you very much for detailed explanation.

My app is a hobby open-source project developed in free time as well. It's still in research stage but slowly grows to something usable. There are no time lines or target dates.

Totally understand your free time issue. No pressure at all.

Lomse is unique as it's the only open-source library for displaying scores. I don't even talk about editing.

I hope the "positioning by measure/beat" is a feature which can be useful for other applications too.

Perhaps, before starting playback the app should inform lomse about what is a beat (i.e a quarter note, a half note, a dotted quarter note, etc.).

In Yamaha's app user selects MIDI file and the app analyses it and displays score. Converting MIDI file into a score representation is a complex task on its own. I've found an "elegant" solution: my app will require user to prepare MusicXML file and put it near MIDI-file. I use MuseScore app for that. Maybe in some distant future my app can convert MIDI to score automatically using portions of code from MuseScore.

Does MusicXML contain information necessary to determine beats? My app has only MIDI-file and MusicXML (converted from that MIDI), it doesn't have any extra info to help Lomse with beats.

Something not related

I also would like measure numbers to be displayed (like on the screenshot). I'll try to add that to Lomse myself. Since they are displayed on constant positions and (hopefully) don't require layouting that can be simple enough for me.

Thanks again for your hard work!

Glad to know that you are not in a hurry with this!

I hope the "positioning by measure/beat" is a feature which can be useful for other applications too.

Yes, for sure. In fact issue #145 shows the need to synchronize the View and playback highlight when an external player is used. And the W3C draft standard (MNX, in development) for music scores also foresee the need to synchronize the semantic representation with real playback instances (i.e. a real performance recorded in mp3). As I'm working on this and it is my aim than Lomse could support MNX standard, both of your requirements (scrolling the view and controlling highlight from the app) should have to be implemented.

Does MusicXML contain information necessary to determine beats?

Yes, and Lomse also does it. But Lomse uses this information for playback, to modulate volume. It is computed on the fly during playback and it is not stored. And, of course, the graphic model knows nothing about beats. Computing beats is not the problem as long as the score has time signature. The real issue is that I need to study how to capture and represent in Lomse the notion of beat so that it can be used for scrolling and for other purposes that require to use it. It would be of help to identify scenarios in which the beat concept can be used. But a fallback mechanism should be designed for scores without time signature; probably just associate beats to quarter notes.

In any case, and with independence of the solution, it could be a feature of your app that the user could set not only the tempo but the figure (note type) associated to a beat, so that the student can choose to practice subdividing the tempo in difficult passages, or doubling it when the musical feeling requires it.

I also would like measure numbers to be displayed (like on the screenshot). I'll try to add that to Lomse myself. Since they are displayed on constant positions and (hopefully) don't require layouting that can be simple enough for me.

Determining the position should be simple, as it is always a fixed position relative to the start of the measure. But coding this in Lomse requires creating several classes, creating its instances in the appropriate places and invoking its methods in the right places. It is not complex, but you will need a good understanding of Lomse internals before being able to do it. I can guide you and give you help and support for this task. But unfortunately numbers that the human engraver could have explicitly assigned to the measures are not yet captured in the internal model so the only option for now is to count measures and display this number. Counted numbers should coincide with those assigned by human engravers in most cases. But there are special cases in which this is not true.

Perhaps it would be simple to start by layouting other needed music symbols, such as pedal marks. The process is similar but you will just copy/paste files, rename classes and adapt its methods.

I will start then with the scroll based on measure/beat issue.

This issue has been split in three (#166, #167 & #168), so I close this one.

lenmus / lomse