[myank] Chord at end of range converts to note

[craigsapp]

In myank, chords turn into single notes when extracting a range of measures.

Example input data:

https://verovio.humdrum.org?t=KiprZXJuCipNNC80Cj0xCjFjIDFlIDFnCj0yCjFkIDFmIDFhCj0zCjFlIDFnIDFiCj00CjFmIDFhIDFjYwo9NQoxZyAxYiAxZGQKPT0KKi0K

**kern
*M4/4
=1
1c 1e 1g
=2
1d 1f 1a
=3
1e 1g 1b
=4
1f 1a 1cc
=5
1g 1b 1dd
==
*-

running the filter myank -m 2-3:

https://verovio.humdrum.org/?filter=myank%20-m%202-3&t=KiprZXJuCipNNC80Cj0xCjFjIDFlIDFnCj0yCjFkIDFmIDFhCj0zCjFlIDFnIDFiCj00CjFmIDFhIDFjYwo9NQoxZyAxYiAxZGQKPT0KKi0K

The third measure only contains a single E4 and is missing G4 and B4.

Only the first note in the token is processed (so not pitch related):

Extracting a single measure does not have a problem:

[craigsapp]

Also ties disappear at the end of an extracted range:

https://verovio.humdrum.org?t=KiprZXJuCipNNC80Cj0xCjFjIDFlIDFnCj0yCjFkIDFmIDFhCj0zClsxZSBbMWcgWzFiCj00CjFlXSAxZ10gMWJdCj01CjFnIDFiIDFkZAo9PQoqLQo=

**kern
*M4/4
=1
1c 1e 1g
=2
1d 1f 1a
=3
[1e [1g [1b
=4
1e] 1g] 1b]
=5
1g 1b 1dd
==
*-

Probably other elements such as articulations will be removed as well. In any case, when a full measure is extracted at the end of a range, the extra processing that you (@WolfgangDrescher) are doing to select partial measures should be disabled (full measure extraction does not need any data editing at the end of a range other than if the ending note(s) have durations that extend past the extracted ending barline.).

[WolfgangDrescher]

This happens due to hre.search(resolvedToken, "([rRA-Ga-gxyXYn#-]+)") in Tool_myank::printDataLine:

HumNum dur = lastLineDurationsFromNoteStart[i];
string recip = Convert::durationToRecip(dur);
string pitch;
HumRegex hre;
if (hre.search(resolvedToken, "([rRA-Ga-gxyXYn#-]+)")) {
    pitch = hre.getMatch(1);
}
string tokenText;
if (resolvedToken->getDuration() > dur) {
    tokenText += "[";
}
tokenText += recip + pitch;
token->setText(tokenText);
lineChange = true;

I had similar issues as you when I lost terminal long notes in myanks because the ll is ignored by the regex. I think we should not parse the pitch but pare the duration related characters and keep all other characters. Additionally we should add support for subtokens.

Am I missing rhythm related characters if I strip away [0-9\.] of the token and then manually add the duration in front of the (sub-) token strings?

I will try to make a PR in the following days.

---

In any case, when a full measure is extracted at the end of a range, the extra processing that you (@WolfgangDrescher) are doing to select partial measures should be disabled (full measure extraction does not need any data editing at the end of a range other than if the ending note(s) have durations that extend past the extracted ending barline.).

This will cause problems with the verovio rendering when the last bar of the example has overfilling noted:

Have a look at measure 6 of this score (Bassus):

https://verovio.humdrum.org/?file=https://raw.githubusercontent.com/WolfgangDrescher/lassus-geistliche-psalmen/master/kern/06-domine-ne-in-furore.krn

**kern  **text  **kern  **text  **kern  **text
*staff3 *staff3 *staff2 *staff2 *staff1 *staff1
*Ivox   *   *Ivox   *   *Ivox   *
*I"Bassus   *   *I"Tenor    *   *I"Cantus   *
*I'B    *   *I'T    *   *I'C    *
*clefC4 *   *clefC3 *   *clefC1 *
*k[]    *   *k[]    *   *k[]    *
*e:phr  *   *e:phr  *   *e:phr  *
*M2/1   *   *M2/1   *   *M2/1   *
*met(C|)    *   *met(C|)    *   *met(C|)    *
*MM180  *   *MM180  *   *MM180  *
=1  =1  =1  =1  =1  =1
0r  .   1A  Straff  00r .
.   .   2A  mich    .   .
.   .   2A  Herr    .   .
=2  =2  =2  =2  =2  =2
2r  .   2c  nit .   .
2E  Straff  2c  im  .   .
2E  mich    4B  ei- .   .
.   .   4A  .   .   .
2E  Herr    4G  .   .   .
.   .   4A  .   .   .
=3  =3  =3  =3  =3  =3
2G  nicht   4B  .   0r  .
.   .   4c  .   .   .
2G  im  2.d .   .   .
2F  ei- .   .   .   .
.   .   4c  .   .   .
2F  -ffer-  4A  .   .   .
.   .   4B  .   .   .
=4  =4  =4  =4  =4  =4
1E  -mut/   4c  .   1e  Straff
.   .   4B  .   .   .
.   .   4c  .   .   .
.   .   4G  .   .   .
1r  .   4A  .   2e  mich
.   .   4B  .   .   .
.   .   2.c .   2e  Herr
=5  =5  =5  =5  =5  =5
0r  .   .   .   2g  nit
.   .   4B  .   .   .
.   .   1e  .   2g  im
.   .   .   .   2f  ei-
.   .   2d  -fer-   2f  -ffer-
=6  =6  =6  =6  =6  =6
2r  .   2.g -mut    1e  -mut/
2E  wañ .   .   .   .
.   .   4f  .   .   .
2A  dein    2e  .   2r  .
2A. zorn    2c  wenn    2e  Wenn
==  ==  ==  ==  ==  ==
*-  *-  *-  *-  *-  *-

VHV does not display such an invalid score. However verovio will actually render it like this:

I'm not sure if this a bug in verovio or if it is humdrum related.

So I think it's good that myank will calculate a valid duration for such overfilled notes. However it should of course keep subtokens and not touch anything of the token beside rhythm related characters.

[WolfgangDrescher]

Off topic but I ran into it while debugging this issue: We talked about this briefly somewhere else, but running make library && make myank takes about 45 seconds for me using an Apple MacBook Pro Silicon M1 "armed to the teeth". So my computer cannot really be the problem. I see that my CPU is not very busy while compiling. Maybe I just have a bad configuration? Or I'm just "spoiled" because I was only using interpreted programming languages before…

Could something like make -j 8 be used?

[craigsapp]

This will cause problems with the verovio rendering when the last bar of the example has overfilling noted

Yes, that is related to my parenthetical comment:

(full measure extraction does not need any data editing at the end of a range other than if the ending note(s) have durations that extend past the extracted ending barline.)

In these cases, the duration of the notes extending past the end of the measure must be truncated. I do not allow under/over filling of measures when parsing Humdrum scores. In Humlib, the overfilling of measures is allowed, with the understanding that the overfill extends into the following measure, as you are doing in the full example. But at the end of the music, all parts must end together. This is for error checking: Humdrum is for analysis as its primary scope. Allowing over/underfilled measures most likely means a data error rather than something intentional, so it is not allowed. It is allowed in Humdrum in some sense, but not in any of my parsers because of the error likelyhood.

Verovio does not check for over/underfilling directly, I think. But it will assign the duration of the measure to the longest part's measure (so verovio always assumes underfilling). In Finale, there can be under- or overfilling since the duration of the measure is set to the time signature. Musescores behaves intermediate to Finale and verovio: it will add invisible rests to pad the measure to the longest part. But it will put an invisible plus sign above the staff warn you that it has added some padding rest, or the duration of the measure does not match the time signature.

What you want to do in this situations (there are also other styles such as cutting the note to the duration of the measure and adding a hanging tie) is to add a layout instruction for the note to indicate that it has a different visual duration from the logical one:

!LO:N:vis=2.
2A

This means that the duration is a half note, but it looks like a dotted-half note.

Here is the full example:

Click to view Humdrum data for above example

```tsv **kern **text **kern **text **kern **text *staff3 *staff3 *staff2 *staff2 *staff1 *staff1 *Ivox * *Ivox * *Ivox * *I"Bassus * *I"Tenor * *I"Cantus * *I'B * *I'T * *I'C * *clefC4 * *clefC3 * *clefC1 * *k[] * *k[] * *k[] * *e:phr * *e:phr * *e:phr * *M2/1 * *M2/1 * *M2/1 * *met(C|) * *met(C|) * *met(C|) * *MM180 * *MM180 * *MM180 * =1 =1 =1 =1 =1 =1 0r . 1A Straff 00r . . . 2A mich . . . . 2A Herr . . =2 =2 =2 =2 =2 =2 2r . 2c nit . . 2E Straff 2c im . . 2E mich 4B ei- . . . . 4A . . . 2E Herr 4G . . . . . 4A . . . =3 =3 =3 =3 =3 =3 2G nicht 4B . 0r . . . 4c . . . 2G im 2.d . . . 2F ei- . . . . . . 4c . . . 2F -ffer- 4A . . . . . 4B . . . =4 =4 =4 =4 =4 =4 1E -mut/ 4c . 1e Straff . . 4B . . . . . 4c . . . . . 4G . . . 1r . 4A . 2e mich . . 4B . . . . . 2.c . 2e Herr =5 =5 =5 =5 =5 =5 0r . . . 2g nit . . 4B . . . . . 1e . 2g im . . . . 2f ei- . . 2d -fer- 2f -ffer- =6 =6 =6 =6 =6 =6 2r . 2.g -mut 1e -mut/ 2E wañ . . . . . . 4f . . . 2A dein 2e . 2r . !LO:N:vis=2. ! ! ! ! ! 2A zorn 2c wenn 2e Wenn =- =- = = = = *- *- *- *- *- *- ```

Another possibility is to leave the dotted-half note as it is, and then add filler invisible rests:

Click to view Humdrum data for above example.

```tsv **kern **text **kern **text **kern **text *staff3 *staff3 *staff2 *staff2 *staff1 *staff1 *Ivox * *Ivox * *Ivox * *I"Bassus * *I"Tenor * *I"Cantus * *I'B * *I'T * *I'C * *clefC4 * *clefC3 * *clefC1 * *k[] * *k[] * *k[] * *e:phr * *e:phr * *e:phr * *M2/1 * *M2/1 * *M2/1 * *met(C|) * *met(C|) * *met(C|) * *MM180 * *MM180 * *MM180 * =1 =1 =1 =1 =1 =1 0r . 1A Straff 00r . . . 2A mich . . . . 2A Herr . . =2 =2 =2 =2 =2 =2 2r . 2c nit . . 2E Straff 2c im . . 2E mich 4B ei- . . . . 4A . . . 2E Herr 4G . . . . . 4A . . . =3 =3 =3 =3 =3 =3 2G nicht 4B . 0r . . . 4c . . . 2G im 2.d . . . 2F ei- . . . . . . 4c . . . 2F -ffer- 4A . . . . . 4B . . . =4 =4 =4 =4 =4 =4 1E -mut/ 4c . 1e Straff . . 4B . . . . . 4c . . . . . 4G . . . 1r . 4A . 2e mich . . 4B . . . . . 2.c . 2e Herr =5 =5 =5 =5 =5 =5 0r . . . 2g nit . . 4B . . . . . 1e . 2g im . . . . 2f ei- . . 2d -fer- 2f -ffer- =6 =6 =6 =6 =6 =6 2r . 2.g -mut 1e -mut/ 2E wañ . . . . . . 4f . . . 2A dein 2e . 2r . 2A. zorn 2c wenn 2e Wenn =- =- = = = = . . 4ryy . 4ryy . *- *- *- *- *- *- ```

In this case the overfilling in one measure is counterbalanced with extra padding after the barline in the other parts. Alternately you could add the padding rests inside of the barline:

2A. zorn    2c  wenn    2e  Wenn
.   .   4ryy    .   4ryy    .
=-  =-  =   =   =   =
*-  *-  *-  *-  *-  *-

This would visually look like the first example, but no need for a LO:N:vis=2. line:

Visually this is close, although the padding rests to fill in the underflow are causing the measure to be slightly wider than in the first example.

(These solutions were discussed in the initial myank discussion in some other issue).

[craigsapp]

Note that the **kern token 2A. is illegal in Humdrum. The 2 and the . should be adjacent. I do permit this in most (but maybe not all of humlib and Humdrum Extras) file processing and analyses, so it is allowed when rendering to notation with Verovio. The problem is that the **recip value is split by non-recip data (the A). This would be somewhat similar to C#C in the pitch representation where the accidental data interrupts the diatonic pitch class data.

2.A and A2. are allowed in **kern, although the traditional (and canonical) ordering is 2.A.

.2A is perhaps permitted (and probably most and maybe all of my parsers for Humdrum can handle it.

[craigsapp]

Am I missing rhythm related characters if I strip away [0-9.] of the token and then manually add the duration in front of the (sub-) token strings?

Yes, you are missing % which is used to encode non-integer divisions of a whole note (other than use of augmentation dots).

For example: triplet whole notes are 2/3rds of a whole note, so the **recip rhythm is 3%2 (3/2 of a triplet whole note equals a whole note):

**kern  **kern
3%2c    1cc
3%2c    .
.   1cc
3%2c    .
1c  1cc
=   =
*-  *-

[craigsapp]

HumNum dur = lastLineDurationsFromNoteStart[i];
string recip = Convert::durationToRecip(dur);
string pitch;
HumRegex hre;
if (hre.search(resolvedToken, "([rRA-Ga-gxyXYn#-]+)")) {
    pitch = hre.getMatch(1);
}
string tokenText;
if (resolvedToken->getDuration() > dur) {
    tokenText += "[";
}
tokenText += recip + pitch;
token->setText(tokenText);
lineChange = true;

Here is the way this code should be rewritten (might be some rough spots as I have not compiled or tested it):

HumNum dur = lastLineDurationsFromNoteStart[i];
string recip = Convert::durationToRecip(dur);

HumRegex hre;
if (resolvedToken->getDuration() > dur) {
        string tokenText = *token;
        // Split the **kern token into separate notes:
        vector<string> pieces = token->getSubtokens();
        // Change the duration of each note:
        for (int i=0; i<(int)pieces.size(); i++) {
                string recipRE = R"re(([\d%.]+))re";
                if (hre.search(pieces[i], recipRE) {
                        string before = hre.getPrefix();
                        string after = hre.getSuffix();
                        // Remove any residual **recip characters after match
                        // due to split **recip data:
                        hre.replaceDesctructive(after, "", recipRE, "g");
                        string newpiece;
                        // Add a tie start if not already in a tie group:
                        if (!hre.search(pieces[i], "[_[]")) {
                                newpiece += "[";
                        }
                        // Replace the old duration with the clipped one:
                        newpiece += before + recip + after;
                }
        }
        string outText;
        for (int i=0; i<(int)pieces.size(); i++) {
                outText += pieces[i];
                if (i < (int)pieces.size() - 1) {
                        outText += " ";
                }
        }
        token->setText(outText);
        lineChange = true;
}

Algorithm in English(somewhat):

If a **kern note/rest at the end of the selection exceeds the duration of the barline, then its duration will be truncated.

To do that, the duration of the note before the barline will replace the original duration to remove the overflow duration from the note.

When the note has an overflow, then a tie start will be added to the note, unless it is already in a tie group.

To update the duration of the note, assume that it is a chord and split into separate single-note strings.

Replace the old duration with the new one, but remove any non-contiguous **recip data in the string after the first **recip character(s).

Note that there is a shorthand for rhythms on chords that I allow (and which I think I have seen you use):

4c e g

which is shorthand for

4c 4e 4g

The above algorithm should be able to handle such shorthands since the e and g notes will not have their durations altered by the above processing.

The above code also fixes cases such as:

4A.

Since the before string will be "", the rhythm will be 4 and the after will be A., then the after string will have any **recip characters removed, so it becomes A without the augmentation dot.

[craigsapp]

I'm not sure if this a bug in verovio or if it is humdrum related.

I haven't fully tracked down such cases, but when I do, it has always been a data error.

The problem is that a null token has had some non-numeric text appended to it. This makes the null token no longer a null token, and the null is interpreted as an augmentation dot. But there is no number part of the rhythm, so a very strange rhythm is generated when converting to MEI/verovio which results in the notation going crazy.

So since it is the result of a data error, I haven't tried to fix it (but it can be difficult to find the error, particularly if you don't know the likely cause of the funny notation).

[craigsapp]

Off topic but I ran into it while debugging this issue: We talked about this briefly somewhere else, but running make library && make myank takes about 45 seconds for me using an Apple MacBook Pro Silicon M1 "armed to the teeth". So my computer cannot really be the problem. I see that my CPU is not very busy while compiling. Maybe I just have a bad configuration? Or I'm just "spoiled" because I was only using interpreted programming languages before…

Could something like make -j 8 be used?

-j 8 might be a good option to add (to allow up to 8 parallel processes for compiling code).

The root of the problem is how I organized the makefile(s). The original intent of humlib was to insert into verovio, and I wanted to do that as a single humlib.cpp file and single include humlib.h. So I have a script that combines all of the individual files into these combined files and then I compile humlib.cpp to create the library in lib/libhumlib.a. And this library file is used to compile the cli programs. This system worked reasonably well when humlib was small, but the compiling process gets longer and longer as humlib.cpp gets larger.

A better solution would be to rewrite the makefiles in humlib so that humlib.cpp is not used within humlib, but rather all of the other src/*.cpp files. This would allow recompiling A better solution (which can also use -j 8) would be either (1) compile src/.cpp into `obj/.oand compilecliprograms with thoseobjfiles, or to create thelib/libhumlib.afiles fromobj/*.o(excludinghumlib.o), and then compile thehumlib` cli programs using that more standard way of creating the library file.

Probably a new makefile called makefile-combined should be created to compile humlib.o by itself. I need to do this to ensure that the combined file compiles correctly for use in verovio. Then I can remove mention of it in the standard makefiles. The humlib.cpp and humlib.h files should then be stored in a different directory, such as humlib/combined. This is the easiest method, although excluding humlib.cpp from the list of files to combine but keep it in src might also work.

There is currently an intermediate solution: make lib will compile the individual src/*.cpp files. I use this to make sure they compile in isolation as well as in src/humlib.cpp. But the resulting compile stops at obj/*.o I think. If this progresses to creating the library file lib/libhumlib.a, then I think this might also be a solution. In that case the short-time compiling method would be:

make lib && make programs

the make lib target does not compile pugixml, so that would also need to be added to make lib.

make library compiles humlib.cpp and then creates lib/libhumlib.a (and also deals with compiling lib/libpugixml.a).

[craigsapp]

Regarding this highlighting box:

How are you implementing it? There is another issue where you commented that I insert such highlighting into the SVG rather than showing with an HTML-based solution. The benefit of inserting into the SVG rather than as an HTML element is that a PDF file created from the SVG will contain the highlighting, which the HTML solution will not be convertible into a PDF (with PDFkit).

Also note that you are overlaying the box. This causes the notes inside of the box to become reddish. Better would be to place the box under the notation so that it does not affect the color of the notation that it overlaps with.

Ideally you would enhance the basic code that I have for this in the VHV repository:

https://github.com/humdrum-tools/verovio-humdrum-viewer/blob/gh-pages/_includes/vhv-scripts/highlight.js

I call the class HnpMarkup since I want to eventually move it to the Humdrum Notation Plugin repo: https://github.com/humdrum-tools/humdrum-notation-plugin

Related to this, I will probably add an issue to verovio repo to have SVGs generated by verovio have some place holders such as g.overlay and g.underlay so that I do not have to insert them myself (and that would allow a standardized way of adding analytic markup to the SVG output from verovio).

[WolfgangDrescher]

Note that the **kern token 2A. is illegal in Humdrum. The 2 and the . should be adjacent.

Thank you I fixed this in my files. Is there a tool to check for a valid humdrum syntax? the humdrum binary (original humdrum-tool) is not helping a lot for such cases.

[WolfgangDrescher]

How are you implementing it?

This is actually just a box I added to the screenshot with the Apple Preview app. But I have implemented this feature now in several tools. The last one being my tricinium project when you navigate to the "Kern" tab of a tricinium and click on the tokens in the code editor. Or the cadence highlighting. Basically a MutationObserver listens to changes in the svg (when verovio updates the score) and then recalculate invisible boxes for each .note, .rest and .verse to improve the clickability (similar to CSS pointer-events: bounding-box; but for all browsers):

• https://github.com/WolfgangDrescher/lassus-tricinium-project/blob/54fe91d9dbc3b6aba9d615f042e9ccf6b7b1fb39/src/components/Humdrum/InteractiveScore.client.vue#L172-L237

For the box highlighting to can have a look at those components:

• https://github.com/WolfgangDrescher/lassus-tricinium-project/blob/master/src/components/Humdrum/NoteHightlight.client.js
• https://github.com/WolfgangDrescher/lassus-tricinium-project/blob/master/src/components/Humdrum/CadenceHighlight.client.js

The benefit of inserting into the SVG rather than as an HTML element is that a PDF file created from the SVG will contain the highlighting, which the HTML solution will not be convertible into a PDF (with PDFkit).

Is this currently working in VHV? When I click "Save as PDF" the selections will not be added.

[WolfgangDrescher]

Have a look at my PR. For now it does not handle overfilling measures with hidden rests, LO:N:vis or dotted barlines but only cutting the notes to the required duration and adding ties. This should work at the beginning and the end of a myanked section. Ideally the methods that we discussed here and in https://github.com/humdrum-tools/verovio-humdrum-viewer/issues/780 should be implemented with an new option to myank. For now I'm happy with the current solution but I can implement this when someone needs it.

[craigsapp]

Is this currently working in VHV? When I click "Save as PDF" the selections will not be added.

It must not be implemented yet. I also use a MutationObserver to watch the SVG on the main page. This should work when downloading SVG images from the File menu on VHV. But for PDF files, I generate them directly from SVG images created with verovio, so they are never placed on the VHV page, so the markup is currently not being applied to the SVG images before they are converted to PDF (I should fix that).

[craigsapp]

Note that the **kern token 2A. is illegal in Humdrum. The 2 and the . should be adjacent.

Thank you I fixed this in my files. Is there a tool to check for a valid humdrum syntax? the humdrum binary (original humdrum-tool) is not helping a lot for such cases.

By illegal, I mostly mean aesthetically illegal. **kern has two main components of pitch and rhythm, so it is not good to break up these two components by interleaving them. **recip is the representation system for (reciprocal) rhythms, so in this case **recip data is being fragmented in multiple locations in the token. This causes problems when doing regular-expression extract of rhythm (or pitch). In this sense it is illegal since it prevents being able to use regular expressions properly (or efficiently).

Here is a webpage that lists the "canonical" ordering of parameters in a **kern token:

https://www.humdrum.org/Humdrum/representations/kern.html

Add to this % for extended rhythm representations such as triplet whole notes.

I also have mostly co-opted < and > for above/below indications for music notation rendering, so they behave similar to x and y and are postfixed the the parameter they apply to. But this feature is only turned on (I hope), when RDF records are given for them in this form:

!!!RDF**kern: > = above
!!!RDF**kern: < = below

A tool/filter that adjusts **kern data tokens to match the canonical form would be good (I don't recall any such tool existing). When I encode data, I cannot remember the full canonical order, so I don't really care (particularly for articulations). But keeping pitch and rhythm parameters grouped is important.

I have a tool called prettystar (in Humdrum Extras) that adjusts interpretation tokens to get similar interpretations in different parts to group onto the same data line. prettykern might be a good name for a **kern canonization tool/filter.

http://extras.humdrum.org/man/prettystar

Also somewhat related is the chord tool/filter, which as you can see is well-documented: https://doc.verovio.humdrum.org/filter/chord Anyway, here are the options:

u|sort-upwards=b    sort notes by lowest first in chord
d|sort-downwards=b  sort notes by highest first in chord
t|top-note=b    extract top note of chords
b|bottom-note=b extract bottom note of chords
f|first-note=b  extract first note of chords
p|primary=b place prefix/suffix/beams on first note in chord
l|last-note=b   extract last note of chords
s|spine=i:-1    spine to process (indexed from 1)
m|minimize=b    minimize chords
M|maximize=b    maximize chords