Add playback support for dynamics and hairpins in voices and specific staves

[avvvvve]

Task description

Currently, dynamics and hairpins affect the entire track and cannot be pinpointed to specific voices or staves.

Users should be able to specify the scope of voices and staves they want dynamics and hairpins to apply to on any instrument. This includes:

• Applying to an individual voice on one staff
• Applying to all voices on the entire instrument
• Applying to all voices on one staff of a multi-staff instrument

To do this, we're introducing a new 'Apply to voice' property for dynamics and hairpins.

Related tasks

The engraving/UI portion of this work will be completed in #22175.

This task will resolve #16812. Also see related (but closed) task #13326.

[Jojo-Schmitz]

See also https://musescore.org/en/node/58031, where this had been requested quite a while ago (in April 2015)

[MarcSabatella]

I would suggest that the default should be for dynamics to apply to all voices. That's the norm for piano, organ, guitar, choral music, etc. Users should not need an extra (and non-obvious) step for the usual case.

[Jojo-Schmitz]

Exactly my point in https://github.com/musescore/MuseScore/pull/19289/files#r1349964296 On top to compatibility with older scores

[bkunda]

I would suggest that the default should be for dynamics to apply to all voices. That's the norm for piano, organ, guitar, choral music, etc. Users should not need an extra (and non-obvious) step for the usual case.

We had deliberated over this. @Tantacrul @avvvvve and I went over the design possibilities many times to arrive at the current behaviour. Indeed it was a particularly hard nut to crack!

My only concern with this suggestion is the potential awkwardness for existing users who could reasonably be perplexed when they see all their dynamics suddenly appearing in purple by default. Those who dig deeper will learn that it applies to “all voices”, but if they only have one voice in their score, it'll just seem confusing. This could be especially strange for anyone writing single-voice music for any non-polyphonic instrument (incl. all ensemble/orchestral scores).

Basically, in introducing the concept of dynamics-on-individual-voices, we’ve created two paradigms: one where dynamics apply to specific voices, and one where they apply to “all voices”. One of these has to be the norm (default), and the other the exception to the norm. We have to make a choice either way.

For the large number of scores that only contain one voice on a stave, the current behaviour (4.1.1) will actually feel the same, because any dynamics applied to that voice will only affect that voice.

But I do appreciate that it will add an extra step when writing multiple voices for polyphonic instruments.

If it ends up causing consternation during testing, we could potentially investigate a preferences setting that lets the user choose which of these "paradigms" should be their default. At least this way, the user will have to discover and initiate the change themselves, so they'll know what's going on!

[Jojo-Schmitz]

One of these has to be the norm (default), and the other the exception to the norm. We have to make a choice either way.

Of course, but IMHO you picked the wrong one.

For the large number of scores that only contain one voice on a stave, the current behaviour (4.1.1) will actually feel the same, because any dynamics applied to that voice will only affect that voice.

That'd be the case if dynamcis default to all voices too

But I do appreciate that it will add an extra step when writing multiple voices for polyphonic instruments.

If it ends up causing consternation during testing, we could potentially investigate a preferences setting that lets the user choose which of these "paradigms" should be their default. At least this way, the user will have to discover and initiate the change themselves, so they'll know what's going on!

How about making all voices the default for pre-4.2 scores?

[bkunda]

@Jojo-Schmitz it is in the design to make "all voices" the default for pre-4.2 scores, so we don't bork their playback.

Good to have this feedback though. We'll continue to gather more of it as we head into the testing period and, if necessary, make a change to the default behaviour.

[Jojo-Schmitz]

OK, that backwards compatibility is good and the reason for my involvement here, so I can happily rest my case ;-)

[MarcSabatella]

I'm less concerned about the extra step (although it certainly is a factor); I'm more concerned about the fact people are likely to simply see dynamics as broken because they don't know about the extra step. I fear this will lead to lots of spurious bug reports and complaints that we will then need to answer. Whereas if the worst that happens with the other choice of default is some questions about "why are my dynamics purple?", that seems less problematic. But I guess we'll see.

Whichever way the default is, I'm not super keen on the idea of a preference to control the behavior, though. We've settled for that approach a few times before (like when we first introduced chord symbol playback), and the result has mostly been extra complication and further confusion. What I personally might do is create custom dynamics already set to all voices, and add those to my palette to use in place of the standard ones (since I expect to want the "all voices" behavior 99% of the time).

I haven't seen the design spec or had the chance to experiment with the PR, but I do have an idea that seems like it would make for a simple enhancement:

What if the default were "all voices", but we used the model of holding Ctrl while adding to mean, apply to single voice only? That's already the established way of adding a key or time signature or barline that is local to a single staff. Of course, we'd keep Properties settings and everything else about the current design.

Of course it it could also be the other way around - single voice by default, Ctrl to apply to all voices. But it wouldn't really solve the surprise problem, and it would feel backwards compared to how Ctrl works in other cases (limiting, not expanding, the range).

[bkunda]

Agree with you Marc about avoiding the path of having different default behaviours in preferences. I suggested this only if we encountered significant division among users during testing.

And I'd really rather avoid introducing concepts like holding modifier keys for this kind of behaviour. It adds complexity to an interaction that really ought to be super simple.

My thinking at the moment is that we need to validate the current behaviour through user testing, which we will be able to do more effectively once it is merged. If we need to flip the default behaviour because the current approach is proving to be problematic, then we can certainly consider that.

[Jojo-Schmitz]

Now a potential fix is in #19866

[avvvvve]

@Jojo-Schmitz Did you mean to comment on #19486?

[Jojo-Schmitz]

Yes, sorry

[avvvvve]

@RomanPudashkin I just updated the issue description with the new design for this feature. Please check Slack for a video walkthrough of the playback portion. Thanks!

[Dima-S-Jr]

This applies not only to dynamic designations, but also to technical instructions and espression marks such as pizz., tremolo, legato, espressivo etc. These instructions should also be able to be applied to individual voices.

[bluebear94]

Since applying a dynamic to a single staff is more common that applying it to a single voice, I think the option to apply it to the staff should be accessible with a single click (with the voice options perhaps grouped into one dropdown).

[wizofaus]

Agreed that this is needed - a lot of piano music has accompanying chords or figures mixed together with melodic fragments that need to be brought out, which can't be done nicely currently as far as I can tell. Note that interestingly the mscx format (and presumably the internal data structures once loaded) are quite capable of having dynamics belonging to different voices, and you can modify a mscx file to have a dynamic belonging to voice 2. But it doesn't appear to impact playback. For MS Basic you can tweak the individual velocities but it's hardly ideal and doesn't seem to work for Muse Sounds.

Oh and for the record, definitely agree the normal behaviour should be that dynamics apply to all voices. Note that in principle it's not even just dynamics - two voices on one staff could in principle have separate playing technique or expression directions (i.e. any text element and maybe even line/spanner elements could in principle be voice specific).

[wizofaus]

(and just today I had a need to have arco and pizz. on the same staff, where the string sections are divisi. Currently it seems pizz. is ignored for playback when applied to notes in voices other than voice 1?)

[avvvvve]

Thanks for the comments @Dima-S-Jr @bluebear94 @wizofaus! Happy to report that the default behavior when adding dynamics will indeed be to apply them to all voices on the entire instrument. There will also be an option in preferences to change that behavior, i.e. setting "Apply to voice" to the voice the dynamic is applied to.

From a UI perspective, we could add the "Apply to voice" property to these types of elements:

• Playing technique annotations (arco, pizz., etc.)
• Sound flags (already has the option to specify entire instrument vs. individual staff, but we could replace this with the "Apply to voice" property which already contains these options)

@RomanPudashkin Do you think it's feasible to get the playback effects of playing technique annotations and sound flags working on individual voices for the 4.4 release?

[Dima-S-Jr]

@avvvvve, I don't know if it's realistic to implement this in 4.4, but how would you like the user to have the opportunity to apply several techniques to one voice in addition? (for example: ½ arco, ½ pizz., etc.)

[erinic04]

This feature is completely broken. even when I select the option for the dynamic to only apply to one staff, it still applies to both staves. Additionally, for some reason my decrescendo leading into the dynamics is completely disabling them so that when what I want is the top staff to play pp and bottom staff to play mp, they just end up both playing at fff and completely drowning out everything else that is happening at the time

[erinic04]

https://github.com/user-attachments/assets/06a04066-643a-4b2a-bd5e-e8fe67ea7eed

[avvvvve]

@erinic04 Was this a score you saved in a previous version of MS Studio or did you create it in the nightly or beta? And are you using Muse Sounds in the mixer or something else?

It'd be helpful if you could attach the .mscz file you're working on so we can investigate—you'll have to .zip it to upload it here.

[erinic04]

I made the score in 4.3.1(2?) then imported it into the 4.4 beta, but this part I'm working on didn't exist yet at the time. It's MS Basic for the "strings" instrument.

(I just renamed to add .zip to the end) shown in the video is m. 88 (also dont leak plz it copyrighted lol)

[avvvvve]

Adding .zip won't do the trick. If you're on a Mac, right click the file in Finder and select "Compress [name of file]". On Windows right click it, go to 'Send to', and then select 'Compressed (zipped) folder'. You may want to send this privately if it's copyrighted since Github is publicly viewable! I'm ben_ave on Discord.

FYI @zacjansheski @DmitryArefiev I may send the file to you to take a look at

[cbjeukendrup]

Adding .zip is fine, because it's just to make GitHub accept the file. When you download it, Safari will auto-unzip it, because it thinks it's a real ZIP; but in the case of MuseScore files, that's no problem, because then you end up with an "uncompressed mscx folder", which MuseScore understands too. So, if Safari indeed auto-unzips it, just open the .mscx file from the resulting folder in MuseScore. (And if Safari doesn't auto-unzip it, then you can just rename it again to remove .zip from the name after downloading, so that it ends in mscz again.)

[erinic04]

Yup! (I also removed the file from the message bc it's copyrighted and I dmed it to him directly)

[avvvvve]

Ah, thanks for the tip! Didn't realize you could skirt it that way. MuseScore wouldn't open the .mscz.zip for me on Mac (I don't use Safari).

Anyway, I'm not seeing the issue with the particular passage you sent in the file now either, though I can see other spots where notes aren't playing, like at the p in Synth 1 in measure 107. I'm not sure it has to do with "Voice assignment" though. There's an issue (https://github.com/musescore/MuseScore/issues/23133) logged for notes note playing back after certain decrescendo/dynamic placements that seems similar to what you're experiencing.

[MarcSabatella]

I don’t think the feature is “completely broken” - I’ve used it quite successfully. I think you simply are running to one of the documented limitations as per the announcement: for soundfonts, per-staff dynamics are supported only for non-SND instruments like piano. Per-voice dynamics are supported for all instruments, and per-staff are supported for all instruments in Muse Sounds. At least that’s what I gleaned from what the announcement says and it does seem to match my experience, although I haven’t tested exhaustively.