Interest in automatic piano fingerings?

[blakewest]

Hi, I just stumbled across Music21. It seems super cool! A couple years ago, I wrote algorithms, and a visualizer for automatically taking any MIDI piano piece, and then visualizing a performance of it in 3D. I wrote all that in JS, but I'm handy with Python too. The algorithms for determining optimal piano fingerings would be especially easy to convert into Python. Would this be of use to the Music21 package? If so, how might we go about adding it?

[mscuthbert]

Hi Blake -- I've wanted automatic piano fingering algorithms for a while for music21 and think it'd be a great addition. (I'd love to have visualizations too, but it's a bit out of the scope of the project). There are a couple things that could be done.

I think that the easiest solution would be to have an external module that you maintain that I can point to on the music21 pages. This way it wouldn't need to conform to music21 coding style, etc. and would be an easy uptake. It could also become integrated into music21 either by you taking it on or I can put it on the list of research projects I could have a student take on -- this is likely to be a lot more work though, because I'm becoming extremely conscious of the additional testing/refactoring/recoding for future changes obligations that come with integrating it into the system, so the code would need extremely intensive tests, docs, coding style, etc., that it might take some of the joy away from the project. We can keep talking about this!

[blakewest]

Hey Michael, Thanks for the info. An external module sounds totally reasonable. Are there docs around how to create plugins for Music21? Looking at the main music21 page as an example, I would imagine the goal of this plugin would be to have some API similar to this:

# Import plugin
from music21-contrib-piano_fingerings import piano_fingerings

bwv295 = corpus.parse('bach/bwv295')
# Save notes list
notes = bwv295.recurse().notes
# Run the algorithm on the notes list, and add a finger attribute to each note
piano_fingerings.find_for(notes) 
# Append the fingering to each note in a way that could be shown on the staff. Like the "addLyric" function.
for thisNote in notes:
  thisNote.addFinger(thisNote.finger.piano)
bwv295.show()

Does this seem like a reasonable goal for the API? How do other plugins work?

[blakewest]

Hey @mscuthbert , just wanted to follow up here. Is this how you envision some kind of plugin working? Are there docs on creating plugins?

[mscuthbert]

Hi @blakewest -- sorry to let this slip for so long. I think that everything is really good about what you've suggested -- I think that it'd be best to use camelCase instead of underscore_case to match music21 (even though Python prefers underscore). I think that what would probably be needed is a "Fingerer" class that is passed a Stream object and then runs a fingering search on it. Thanks!

[yipcma]

I'm curious if there's any current plan for fingering integration, or if there's any other stand-alone solution you've noticed :)

[blakewest]

@yipcma I don't have any current plans to put in significant time to building this module. That said, I did the hard work of figuring out reasonable algorithms several years ago, and you can find that code here: https://github.com/blakewest/performer/tree/master/src/Algorithms . So it may not be that hard to re-use them for whatever project you need it for.

[marcomusy]

Hi @mscuthbert @blakewest @yipcma I wrote since quite some time some python code that I have finally put together and just committed. Have a look if you like: https://github.com/marcomusy/piano-fingering Cheers, Marco

[blakewest]

@marcomusy this looks cool. I like that you can change some parameters like hand size. And the visualization part is a nice touch. Looks like a good start. Looking at the fingerings on the bach piece shown on the README, I can see some discrepancies from what the orthodox fingering choice would be. For instance, you have 2,3,5 playing F, A, D. Having 5th finger stretch from third finger from A - D is a considerable distance. The typical choice there would probably be 1,2,5. Similarly, I see the left hand using doing 1,2,1 to play G,A,Bb. Here, using thumb to play the Bb is awkward because of the black key. Each of those would point to some tweaking with the weights you're assigning to each option. I'm also a bit unclear how many path options you're searching. And the following is only trying to be helpful. If you know all this, you can ignore. But when I was writing my algorithms for this before, my general approach was the following...

• Create a "database" of "costs of movement" for every possible finger movement combination for every possible key combo (up to about 1.5 octaves). That is, "from-finger" on "from-note" to "to-finger" on "to-note". For example, C up to E going from thumb to middle finger. And then do that for every possible option. I wrote some functions to calculate what this would be. And those functions had to include special rules for when moving from a white key up to a black key, or black key up down to a white key. And even some sub-special rules for when you're using thumb on those black keys, cause the cost for those is significantly higher than other fingers. This database is pretty annoying, but it really just turns into a big hash and you only have to calculate it once.
• Once you have this database, you ingest the file, and you look at the notes, and in conjunction with your cost database, you have to find the lowest cost path algorithm. I implemented this using Viterbi's Algorithm. I used it cause it actually allows a full search of all options in linear time, rather than exponential time.
• The lowest cost path is the fingering, so you just align it with the notes, and you're good to go.

I'm happy to talk more about this if you're interested. Great work getting started on this!

[marcomusy]

Hi @blakewest, thanks for your comments. I tried to use your javascript but i could not make it work, I'm maybe doing something wrong. My approach is rather straightforward: search only for combinations that are "reasonable" (no crossings of 2345 fingers etc). This is coded in function Hand.skiprules(). This trims a lot of combinations to search through (if i remember it reduces to a few hundreds). I didn't want to use a database because: It's so boring... :) It's not dynamic (the hand is elastic, it somehow remembers past positions, i gave it first try in set_fingers_positions(), but didn't have time to explore it deeper) Doesn't allow for different hand sizes (which i think it's kind of important to consider) Viterbi's algorithm could improve speed but at moment an average score can be processed in <30s. You are right that fingers 1,5 should be discouraged to play black keys, this can be obtained by reducing Hand.bfactor (but I think that in the bach invention is kind of ok). If fact it would be interesting to train the internal parameters (eight in my case), at least on scales or typical patterns or on a bunch of classic scores already fingered by professional pianists.

[mscuthbert]

Looking great! btw -- we're more likely to incorporate something like this into a rep: https://github.com/cuthbertLab/music21-tools -- putting externally created tools there makes it easier to not need to have the same level of day-to-day updating of them (I'll just fix once per release instead of at every commit) and has really made my work on music21 a lot easier to do. :-)

[marcomusy]

Feel free to incorporate it in the tool repo, if you think it is mature enough. Congratulations btw for music21, really impressive piece of software!

[blakewest]

@marcomusy Regarding hand sizes, in my experience of teaching piano, hand size only affects the fingering choice in rare circumstances. Mostly cause if someone's hands are very small (like a child's hand), then they simply don't play songs that require larger hands. And after you're a child, almost everyone can reach an octave, which is in practice enough for 95% of songs. But then, if there was a significant difference, you could create multiple cost databases, just with different weights for each size.

Regarding the 1 and 5 fingers being discouraged from playing black keys... it's only in certain circumstances that this should be true. Indeed, playing octaves on black keys is often done with only 1st and 5th fingers. So if the b-factor is circumstantially aware of when to add the cost and when not to, then you're probably fine. But if it's a global cost, then it will improve some situations, while necessarily making others worse.

Technical implementations aside, the real need is just to have correct fingerings. Bach pieces like you have are great test pieces, because the correct fingerings are well known and freely available. They're also on the easier side. They don't include the complications of how to handle fingerings with pedals, since most bach pieces have little to no pedal work. So if you test it with a few bach pieces and a few other well known pieces, and the accuracy is high (I would think like > 95% the same as official fingerings), then you've likely got a solid algorithm. If not, then I'm happy to help incrementally improve the algorithm until it's great. I say all this only to be helpful in making sure we have a quality module for music21. Cheers - Blake

[marcomusy]

Thanks for your comments, It would be interesting to test it as you say on a set of bach scores which already contain fingering indication. Do you have any link to such fingered scores? Unfortunately it is not easy to assess how "good" a specific choice is, as this remains largely subjective and sometimes motivated by accidental reasons (later repetition of the same pattern, sync w/ left hand, expression etc). I have at hand a good quality edition of fuer elise with a fingering that i'm pretty sure no naive algorithm would ever guess... Without changing the 'philosophy' of the algo, i see a couple of possible ways to improve it:

• allow the hand to dynamically adapt its rest-position based on the notes already played (i.e. playing e4-e5-e4-e5-e4-e5 would theoretically have zero "difficulty" as the fingers are already in place). At the minute the hand moves rigidly. This is very doable, I already wrote a version of set_fingers_positions_hide() that does that, but didnt have the time to develop it.
• add rules to deal with things like thumb crossing, black keys etc
• having a human pianist judging where the algorithm only miserably fails, and manually tweak the internal parameters to correct for those cases without disrupting what works.

I don't think this or any algorithm for fingering will ever be "great" but I think it would become helpful sometimes to suggest possible solutions one would not think of.

[blakewest]

Yeah, just googling "bach invention sheet music with fingerings" will bring up many results. Here's one. Standard scores do not contain fingerings for every single note. But they usually contain the ones where there could be a question, and the rest are assumed to be the finger closest at hand. I would push back on the idea that fingerings are largely subjective. While two high level players might disagree on a few notes within a piece, they would agree on almost all of them, and the rest they would recognize as "another good option". Any good algorithm should be able to be in that ballpark as well. Where even if a high level player disagrees, they would say, "sure I might do it like this, but the algorithm picked another reasonable option."

I think allowing the hand to dynamically adapt it's rest-position will be a great improvement.

Separately, when I was writing my algorithm, I found it very useful to write tests for certain cases. It revealed various flaws, and helped me improve the algorithm.

Best - blake

[marcomusy]

writing a few test cases is actually a very good idea, if you still have those I might reuse them.

[blakewest]

Here's a link to the test folder for my repo: https://github.com/blakewest/performer/tree/master/test It says "costDatabaseSpec", but it's not super well named. If you read the specs, you'll see that it tests various short note paths which have several semi-reasonable fingering options, and ensures that the correct choice has a lower cost than a close, but not as good option. I will say, the "2nd place" option I tested against was fairly arbitrary. And more proper specs would have actually run the algorithm against note paths, rather than just summing up path options from the cost database. But none the less, the specs touch on a number of important edge cases for piano fingerings, and getting all of them right made my algorithm much better.

[jacobtylerwalls]

Not on the roadmap for music21 itself in 2022, but very good to have this search-discoverable discussion of other packages working on this useful feature.