A timeline for displaying music scores

[FelipeDefensor]

It is time to address the elephant in the room, which is displaying musical notation.

Conventional scores and TiLiA use space differently in a fundamental way: horizontal space is always to-scale in relation to time in TiLiA, while in conventional scores that relation is frequently relaxed in order to avoid overlaps or packing symbols too close together. There are no similar constraints in the use of vertical space, on the other hand.

With that in mind, I think that a good solution is to go with a mix between a piano roll and a conventional score, using a to-scale horizontal axis and a staff-based vertical axis. If I didn't believe a picture speaks a thousand words, I wouldn't have made TiLiA, so here's what I mean. This:

Would be displayed in the score timeline as:

Ideally, the data structures will be set up in a way that will also allow for a piano roll and conventional UIs to be easily implemented on top of them, later.

There are some things to be sorted out if we choose to go with this representations, for instance:

• How to display simultaneously seconds? Their rectangles would overlap.
• How to display accidentals when another note would obstruct them?
• Where to display symbols like clefs and time signatures? They would overlap with notes if displayed directly on top of the staff.

We definitely want to be able to import from conventional formats like MusicXML, which will require writing some kind of parser to our internal format. I have not yet researched about it, but I imagine there is already some available Python tooling. In the worst case, we can wrangle the format ourselves.

Another option is to go straight into conventional notation using tools like Verovio. That is something I am not sure I am ready to tackle, due to the issues that probably will arise with the differences in dealing with horizontal space.

[FelipeDefensor]

This was originally a feature that I intended to work on sometime in 2025, but that I decided to implement right now due to the needs of my PhD, of the Sinergia project and of the Anlysis-a-Thon taking place early 2025.

[FelipeDefensor]

@azfoo I don't know if you have given this feature some thought already. Please let me know you opinion. Also, @MarkGotham, you probably will have something to say about this.

[FelipeDefensor]

This issue #218 mentions some features that we want to have in place before releasing the score timeline.

[azfoo]

I'm not sure about displaying music with blocks - it isn't the most readable and there will be a (re)learning curve to reading scores in the app. Perhaps we could go with using the space created between beats to fit in all of the notes and accidentals attached to that beat somewhat equally spaced by their time value and use conventional notation. We could even float the clef, time signature and key signature on the left of the timeline, so that it is always there.

Regarding parsing MusicXML, I've read up a little on the documentation and converting directly from that to TiLiA elements shouldn't be complicated enough to require an external library.

[FelipeDefensor]

I'm not sure about displaying music with blocks - it isn't the most readable and there will be a (re)learning curve to reading scores in the app.

I agree those are tradeoffs to be considered.

Perhaps we could go with using the space created between beats to fit in all of the notes and accidentals attached to that beat somewhat equally spaced by their time value and use conventional notation.

That would be great, but it only works if we are zoomed in quite a lot. The typical zoom levels I'm used to work would result in very ugly overlaps, I imagine. Can we get around that? I was thinking maybe some parallax could do the trick. We keep the score zoomed in so it does not overlap. but scroll it a faster rate than the rest of the timelines. Even then, we lose what might be a helpful zoomed out overview.

We could even float the clef, time signature and key signature on the left of the timeline, so that it is always there.

That's a good idea, but we would still need to figure out a way of showing them aligned with the notes sometimes to signal clef/signature changes.

Regarding parsing MusicXML, I've read up a little on the documentation and converting directly from that to TiLiA elements shouldn't be complicated enough to require an external library.

That's good to hear. If you dig around the score-timeline you can find the attributes I'm currently using and parse with that in mind. They are not complete nor final, of course, feel free to propose changes.

[azfoo]

Perhaps we could go with using the space created between beats to fit in all of the notes and accidentals attached to that beat somewhat equally spaced by their time value and use conventional notation.

That would be great, but it only works if we are zoomed in quite a lot. The typical zoom levels I'm used to work would result in very ugly overlaps, I imagine. Can we get around that? I was thinking maybe some parallax could do the trick. We keep the score zoomed in so it does not overlap. but scroll it a faster rate than the rest of the timelines. Even then, we lose what might be a helpful zoomed out overview.

Alternatively, we could have it as a QDialog that scrolls (something like a dockable PdfViewer but without page numbers)? I get zooming on a piano roll but not on music notation.

[FelipeDefensor]

Alternatively, we could have it as a QDialog that scrolls (something like a dockable PdfViewer but without page numbers)? I get zooming on a piano roll but not on music notation.

It could help with the display, but maybe interaction would become harder as it wouldn't share the QGraphicsScene with the rest of the timelines. Box selecting, for instance, wouldn't work.

The "blocks" are a mixture between conventional notation and piano rolls. One of the good things they would (supposedly) get from piano rolls is precisely remaning meaningful when zoomed out. I should have a prototype be the end of the week, so if you're able to spin up a minimal parser until then we can test with real scores to see how things look.

[azfoo]

I'll hopefully have a functioning parser in the next two days. Just to clarify because I couldn't figure out from the code: what do step and octave in ComponentKinds.CLEF mean?

[MarkGotham]

Hi both, and thanks for this!

Great to have this conversation. Here are some notes and "in principle" opinions.

Let's not assume that a score rolls in the same way as the other, continuous timelines. The current approach of "page turns" works well IMO. Plenty of room for improvement on that, but many have thought about this problem and settled on staying with something like the traditional static score "page" (broadly defined) and controlling page turns. A good example is performers using ipads and similar: there's never a rolling score; always fixed pages and often a device to allow the performed to chose when to turn.

How best to make that work in our case is open to design. One approach is to highlight (e.g., bounding box) the area of the other timelines that's currently being displayed on the score side.

To be clear, I'm not talking about static images in the sense of an imported PDF here. I definitely support the move to semantic score reading and rendering. Have a look at notable precedents for part/all of this equation, e.g., Dezrann, Vexflow, and Verovio.

Speaking of Verovio, conversion of our scores via MEI is potentially a big project, but worth considering for this among other reasons.

Hope that's clear and helps a bit. Happy to chat sometime.

[azfoo]

Could we have conventional notation in a dock widget but in one line that "scrolls" (more like a quick horizontal shift) when the next measure is about to be out of view? The timeline version of this could be a zoomed out image of the same, that somewhat aligns with the beat timeline, with a bounding box indicating the position that the dock widget is showing. The scroll position of the timeline and dock widget can be linked.

That way, we don't actually need to handle making components (which will only make the total file size bigger) and instead render an SVG (maybe hosted online) with a csv on image x-position (start/end) to measure number.

[FelipeDefensor]

I'll hopefully have a functioning parser in the next two days. Just to clarify because I couldn't figure out from the code: what do step and octave in ComponentKinds.CLEF mean?

Step is zero-based numeral correspoding to the "note name": C is 0, D is 1, ..., B is 6. ComponentKinds.Clef is the component kind for clefs. As those will need to be positioned and will also be involved in caclcuting note positions, they will be a component like everything else.

[FelipeDefensor]

Could we have conventional notation in a dock widget but in one line that "scrolls" (more like a quick horizontal shift) when the next measure is about to be out of view? The timeline version of this could be a zoomed out image of the same, that somewhat aligns with the beat timeline, with a bounding box indicating the position that the dock widget is showing. The scroll position of the timeline and dock widget can be linked.

I'm not sure I got that completely, but I think it might work, yes.

That way, we don't actually need to handle making components (which will only make the total file size bigger) and instead render an SVG (maybe hosted online) with a csv on image x-position (start/end) to measure number.

Components are needed to make annotating score elements possible. I think that adding comments and coloring notes, at least, is a feature that we should have. Later, we could go towards having more complex graphical annotation features like circling or bracketing groups of notes, which are features that are very useful for analysis and but that conventional score editors support only poorly.

That being said, maybe we could create components only for the score elements that are annotated.

[FelipeDefensor]

Let's not assume that a score rolls in the same way as the other, continuous timelines. The current approach of "page turns" works well IMO. Plenty of room for improvement on that, but many have thought about this problem and settled on staying with something like the traditional static score "page" (broadly defined) and controlling page turns. A good example is performers using ipads and similar: there's never a rolling score; always fixed pages and often a device to allow the performed to chose when to turn.

I really think we should have some timeline-like visualization, as that will make following the score and connecting the score symbols to the rest of the annotations in TiLiA much easier. I don't think performing a piece is a relevant use case for TiLiA. That said, I don't see why we couldn't have both, and probably the "vertical" score with page breaks will be easier to implement right now.

What about we keep moving to have a piano-roll/score mix like I suggested, one that does not display all the score symbols (as all those would significant more time to implement) but also have a conventional and complete score based view using the bounding box like you suggested by leveraging Verovio or something similar?

[FelipeDefensor]

@azfoo I have just pushed the prototype. Will take a look at engraving tools like Verovio and work on other issues until you're done with the parser.

[azfoo]

@FelipeDefensor mostly done with the parser - it needs a few more bits to fully integrate (see commit)

[azfoo]

@FelipeDefensor parser update: done one more thing that would make things easier is to have a staff id on score components, so that multistaff music can be displayed.

[FelipeDefensor]

@FelipeDefensor parser update: done one more thing that would make things easier is to have a staff id on score components, so that multistaff music can be displayed.

Great. Will take a look at it later today, maybe tomorrow.

Agree about the id. The position attribute is meant for that, I just haven't implemented the UI for that. Here's a very rough list of what's to come if we decide to go forward with this representation:

• How to scale or crop mixed nature components?
• Accidentals
  • Shrink if zoomed out far enough
• Staff
  • Support multiple
• Barline
  • Different barline types
• Clef
  • Validate clefs are not created at the same time
• Time signatures
  • Handle multiple digits
• Key signatures
  • Vary accidentals position according to clef
• Staccatos
• Dynamics
• Grace notes
• Slurs
• Fermatas

[FelipeDefensor]

I think we should explore the conventional notation stuff for a bit before deciding what to prioritize next. I had no luck getting a Verovio SVG working in PyQt. Here's the script I'm trying to run:

import sys

import verovio
from PyQt6.QtGui import QPainter
from PyQt6.QtSvgWidgets import QSvgWidget
from PyQt6.QtWidgets import QApplication, QLabel

tk = verovio.toolkit()
tk.loadFile('example.mei')
svg_string = tk.renderToSVG()

svg_bytes = bytearray(svg_string, encoding='utf-8')

app = QApplication(sys.argv)
svgWidget = QSvgWidget()
svgWidget.load('example.svg')
svgWidget.setGeometry(100, 100, 300, 300)
svgWidget.show()
label = QLabel('dhsjkadhsjak')
label.show()
app.exec()

I'm pretty sure this is supposed to work. Could you try it on your end?

[azfoo]

I'm pretty sure this is supposed to work. Could you try it on your end?

...
tk = verovio.toolkit()
tk.loadFile('example.mei')
svg_string = tk.renderToSVG()

svg_bytes = QByteArray(bytearray(svg_string, encoding='utf-8'))

app = QApplication(sys.argv) svgWidget = QSvgWidget() svgWidget.load(svg_bytes) svgWidget.setGeometry(100, 100, 300, 300) svgWidget.show() app.exec()

this solves some of the issues but I get errors about fontsize. Writing to file gives a valid SVG that can be opened - I suspect this has to do with PyQt.

[azfoo]

update: exporting a file as svg from an engraver shows up fine so maybe it isn't PyQt but verovio

[FelipeDefensor]

update: exporting a file as svg from an engraver shows up fine so maybe it isn't PyQt but verovio

Maybe they use different SVG specs?

[FelipeDefensor]

@FelipeDefensor parser update: done

@azfoo I took a look at it and it seems to work well, except the first measure seems to be skipped when importing.

Here are the .tla and the .mxl to reproduce.

lvb-16bars.zip

[FelipeDefensor]

@azfoo I took a look at it and it seems to work well, except the first measure seems to be skipped when importing.

I removed the anacrusis and now eveything works perfectly. Here's a demo:

https://github.com/user-attachments/assets/94f310f5-8f6a-4ca7-9395-539016383fb3

I believe this kind of visualization (or at least piano roll) is worth our time, while not excluding the obvious value of also having conventional notation.

[azfoo]

@FelipeDefensor I'm about to merge the vexflow viewer into the app. Should this be done as an alternate view to the blocks (on the same timeline, with the option to view/hide) or a different timeline?

[FelipeDefensor]

I like the alternate view idea. Anything against that? Ideally we would separate now and merge later, but there's too much boilerplate to creating a new timeline kind.

[azfoo]

Cool. Will do.