notkaramel commented 4 months ago

From the ECHOES Project report 2023

Continuation of #20

Pitch similarity

to search for specific pitch patterns across the corpus and to gauge the melodic similarity between chants, or portions of chants. However, because we are dealing with chants written in two different music scripts, one more pitch-oriented (square) and the other more interval-oriented (Aquitanian), we must take this into consideration when deciding to what extent the items compared are a good match with the query. For instance, we wish to be able to seek for a specific pitch pattern, e.g. A, G, A, F or for a specific interval pattern, e.g. '2nd up, second down, third up, fifth up'. Ideally, we’d like to be able to search across the whole repertory of chants, or a portion of them (for instance, search the pattern only on some specific sources, or sources with the same kind of musical notation). Obviously, if we seek for a specific pitch pattern, e.g. 'A, G, A, F', this exact pattern could only be found in sources in square notation, while sources in Aquitanian notation can return as a result only a series of intervals, such as a note followed by a note a second lower, one a second higher, and then one a third lower from one or more sources.

Categories of results

A perfect match. When I search for a specific series of pitches and the results bring me exactly the same melody
An almost perfect match. When I search for a specific series of pitches, the results bring me a series of intervals that could match the exact pitches I was looking for, but they are found written in a source in Aquitanian notation.
A very good match. If the two compared versions show the same melody, but a different note distribution among the syllables. Nota bene: we are not interested in the text of the syllables but only on the information regarding how many notes are assigned to one or more syllables.
A good match. If the two compared versions differ for just a few notes (up to 20%, e.g. some notes are missing in one of the two versions or are dislocated at a different height).
A fair match. When the two versions still bear enough resemblance, but they differ up to 40% notes at several points (e.g. melismas, intonation, cadences) and/or one of the two melodies has a lacuna or uncertain reading.
A poor match would be used for melodies that differ by above 40%.

Other results that would fall into the categories (‘good match’ and ‘fair match’) are for compared versions that show the same pattern but with a few extra ones.

Older Report (documentation purposes)

An older version of the report mentioned absolute numbers for the good, fair, and poor match categories rather than percentages. Here I include the original text:

A good match. If the two compared versions differ for just a few notes (up to 4, e.g. some notes are missing in one of the two versions or are dislocated at a different height).

A fair match. When the two versions still bear enough resemblance but they differ up to 8 notes at several points (e.g. melismas, intonation, cadences) and/or, one of the two melodies has a lacuna or uncertain reading.

A poor match would be used for melodies that differ 9 or more notes. Obviously the reference numbers of notes mentioned above (4, 8, and 9 or more) apply for a comparison of a pitch/interval pattern of 20 notes or more. For analysis of shorter patterns, these numbers would be drastically reduced.

I (@martha-thomae) asked Elsa to change these absolute numbers into percentages, but I am including here the original message for reference.

notkaramel commented 4 months ago

Also implement Exact Pitch search to work on Aquitanian chants to get the almost perfect match

martha-thomae commented 4 months ago

A few changes:

We will remove the "Indefinite pitch" search category.
We will change the description of the "Exact pitch" category from this:

Exact pitch (only for Square music script chants): Exact pitches could be (e.g., a b E F). Use uppercase letters for a lower octave and lowercase letters for an upper octave.

To this:

Exact pitch (chants with an identified mode): ...
We will change a bit the results returned by the "perfect" match and the "almost perfect" match, from this:
Categories of results:
- A perfect match. When I search for a specific series of pitches and the results bring me exactly the same melody
- An almost perfect match. When I search for a specific series of pitches, the results bring me a series of intervals that could match the exact pitches I was looking for, but they are found written in a source in Aquitanian notation.
To this:
Categories of results:
- A perfect match. When I search for a specific series of pitches and the results bring me exactly the same melody (for square notation and for Aquitanian notation where the mode is identified, as that would imply that the pitches of the notes are known).
- An almost perfect match. When I search for a specific series of pitches, the results bring me a series of intervals that could match the exact pitches I was looking for, but they are found written in a source in Aquitanian notation for which the mode is not identified ("undetected mode").
From the previous item, the "perfect match" in Aquitanian notation (for the pieces where we know the mode), please consult the next comment.
We will implement the almost perfect match for "undetected mode" in Aquitanian by converting the pitch sequence into a contour sequence. It doesn't matter that the results will have overlapping sequences. that already happens in contour-based search, and that is fine. Example: contour of 1 1 gives the following results (look at the second and fourth groups of highlighted purple notes). Elsa and I think this is fine.

martha-thomae commented 4 months ago

For item 4 in the previous comment, the pitches of the notes in Aquitanian notation can be inferred from the mode (in case the chant has a mode). See the last column of the following table for this (the same information is in here):

IF last note	AND for every nc with tilt="se"	THEN:	comments	pitch at line
loc = -2	loc = -1 OR loc = +3	mode = 1		f
loc = 0	loc = +1 OR loc = -2	mode = 2		d
loc = -2	loc = -2 OR loc = +2	mode = 3		g
loc = -1	loc = -1 OR loc = +3	mode = 4		f
loc = 0	loc = 0 OR loc = +4 OR loc = -3	mode = 4	(rare case, earlier chants)	e
loc = -2	loc = +1 OR loc = -3	mode = 5		a
loc = 0	loc = -1 OR loc = +3	mode = 6		f
loc = -2	loc = 0 OR loc = +3	mode = 7		b
loc = 0	loc = +2 OR loc = -2	mode = 8		g

martha-thomae commented 4 months ago

@notkaramel, sorry, I was sure I left these comments a few days ago, but it seems that wasn't the case.

Could you please check this 30-minute video as well? Especially the last part. If you could take a look at it before the meeting, that would be awesome!

martha-thomae commented 4 months ago

Dear @elsinhadl, based on an older version of the report (found at the end of the original issue), you seem to think that musicologists would search for large sequences of pitches (approx. 20 pitches long), and based on these long sequences you thought about what a good or fair match would look like. Could you provide a few examples of usual sequences an expert would like to look for? Thank you!

martha-thomae commented 4 months ago

@notkaramel:

For the following four categories of results with a different percentage of matches, we will use Smith-Waterman:

perfect match (matches = 100%)
good match (matches >= 80%)
fair match (matches >= 60%)
poor match (everything else)

We were planning on this when we talked about cross-comparison, saying that, eventually, we will need to compare a small sequence (a query) against a large sequence (the chant) when doing the search part of the interface. I think this will be the best approach since we need to find the percentage of matches.

I have been experimenting with this already in Python using the minineedle library and it seems to work. I am including some data below about this to illustrate what I am saying. So, could you start looking into using Smith-Waterman for this? First try to see if there is a library already implemented for JavaScript to use Smith-Waterman (if not, then you would need to implement it).

from minineedle import needle, smith

chant = ['d', 'e', 'f', 'd', 'e', 'd', 'd', 'e', 'd', 'c', 'd', 'f', 'd', 'e', 'f', 'd', 'd', 'f', 'a', 'g', 'a', 'g', 'f', 'g', 'b', 'g', 'a', 'b', 'b', 'a', 'f', 'g', 'a', 'g', 'g', 'd', 'g', 'f', 'f', 'e', 'f', 'd', 'c', 'd', 'e', 'f', 'd', 'a', 'b', 'a', 'g', 'a', 'g', 'a', 'c', 'c']
query = ['d', 'c', 'd', 'e', 'f', 'd', 'f', 'd', 'f', 'a']

alignment = smith.SmithWaterman(chant, query)
alignment.get_score()

The returned result was 6 (six matches). Given the query, which has a length 10, the results would fall in the 6/10 = 60% match. Therefore, this would be a fair match.

The next code is me trying to get the matches by myself. By the way, we could use this part to get a list of the @xml:ids of the matches to later highlight them in Verovio.

store = True
matches = 0
difference = 0

alig1, alig2 = alignment.get_aligned_sequences()
for i in range(0, len(alig1)):
    if (alig1[i] == alig2[i]):
    # match
        if store:
            matches += 1
            # append the xml:id to a list
        else:
            matches = 1
            # RESTART collecting xml:ids on the list
        store = True
    else:
    # mismatch or indel
        difference += 1        
        store = False

print("MATCHES: " + str(matches))
print("DIFFERENCES: " + str(difference))

The result returned:

MATCHES: 6
DIFFERENCES: 41

(If I ran Needleman-Wunsch instead, I would have MATCHES = 10 and DIFFERENCES = 46, giving me the total length of the chant I used, which was 56).

I am adding this folder here for documentation purposes; please ignore it, Antoine. The file includes a Python script to compare the query against the chant in MEI. I parsed the MEI chant into a sequence of pitches (a list of pitches), applied Smith-Waterman using the query (list of pitches) and the parsed list of pitches of the MEI chant, obtained the Smith-Waterman score and matched sequences (with the gaps), and retrieved the @xml:ids corresponding to the matches (this implied adding "None" @xml:id values for the missing notes in the chant with respect to the query [or extra notes in the query]).

Python_queryPitchSeq_overMEIchant.zip

martha-thomae commented 4 months ago

Hmmm, regarding the library for Smith-Waterman, what about this one: https://cdcgov.github.io/bioseq-js/?

notkaramel commented 4 months ago

@martha-thomae I will try to answer all of your comments one by one as soon as possible.

As for Smith-Waterman, I've already implemented the algorithm so maybe we can test it out directly on the source code soon. If it's not good enough, we can start testing with the library that you suggested!

martha-thomae commented 4 months ago

Hi @notkaramel:

As for Smith-Waterman, I've already implemented the algorithm so maybe we can test it out directly on the source code soon. If it's not good enough, we can start testing with the library that you suggested!

This is great, but please make sure to test your implementation first with examples for which you already know the answers (either that you created or you found online with their corresponding answer). You will need to make 100% sure that your implementation is correct, otherwise, we won't be able to know if Smith-Waterman will work for the different melodic matches in our corpus because we won't be sure if the issue is with the methodology or the implementation of said methodology. So, please, make sure the implementation is working as intended before experimenting with the ECHOES corpus. Thank you!

martha-thomae commented 4 months ago

Now, since this is a long issue, I think we need to get to things in a certain order. Here we have the issue:

Categories of results:

(i) A perfect match. When I search for a specific series of pitches and the results bring me exactly the same melody (for square notation and for Aquitanian notation where the mode is identified, as that would imply that the pitches of the notes are known). (ii) An almost perfect match. When I search for a specific series of pitches, the results bring me a series of intervals that could match the exact pitches I was looking for, but they are found written in a source in Aquitanian notation for which the mode is not identified ("undetected mode"). (iii) A very good match. If the two compared versions show the same melody, but a different note distribution among the syllables. Nota bene: we are not interested in the text of the syllables but only on the information regarding how many notes are assigned to one or more syllables. (iv) A good match. If the two compared versions differ for just a few notes (up to 20%, e.g. some notes are missing in one of the two versions or are dislocated at a different height). (v) A fair match. When the two versions still bear enough resemblance, but they differ up to 40% notes at several points (e.g. melismas, intonation, cadences) and/or one of the two melodies has a lacuna or uncertain reading. (vi) A poor match would be used for melodies that differ by above 40%.

Other results that would fall into the categories (‘good match’ and ‘fair match’) are for compared versions that show the same pattern but with a few extra ones.

We will leave two things at the very end (to be dealt with only if time allows), these would be: the thing about having lower and upper octaves allowed in the query, and the "very good match" category (see the last part of this comment for details).

So, we will proceed in the following order starting with the "exact pitches" search: (A) Implement perfect (i), good (iv), fair (v), and poor (vi) matches for "exact pitches" by using Smith-Waterman as discussed before. Limit it in only square notation first as it will already has all the pitches in the @pname attributes of the <nc> elements. (B) Proceed to extract the pitches for the Aquitanian chants that do have a mode according to the last column of this table. We can plan on storing this on a separate place rather than having to parse this every time to have an array of pitches (maybe store it on a csv file for all the Aquitanian chants with mode?) (C) Now test step 1A with these Aquitanian chants that do have a mode (the ones from step 1B).
Now we will move to the almost perfect match category of "exact pitch," which should be a perfect match on a contour-based Aquitanian chant with "undetected mode": (A) When the user enters a pitch-based query, show the contour-based corresponding query at one side of the original query. (B) With the resulting contour-base query of step 2A, find all the almost perfect matches (ii). An almost perfect match based on a pitch sequence would be the equivalent of a perfect match for the correspoding contour sequence. Remember to only look for the perfect match of the contour-based sequence in Aquitanian chants that have mode "undetected".
We will move into "contour-based" search (for all chants, square, aquitanian, with and without mode): (A) Have a completely new text-box for this type of query (different from the pitch-based textbox) (B) Implement perfect (i), good (iv), fair (v), and poor (vi) matches for "contour" by using Smith-Waterman. (This is replicating step 1A, but for a contour sequence rather than an exact-pitch sequence.

The last thing would be to evaluate whether we need to implement the following (depending on the time left):

The change in octaves given by: lower octave encoded as c d e f g a b and upper octave encoded as c' d' e' f' g' a' b'
The very good match that deals with different note distributions. Here is where we would need to solve the "neume boundaries" issue https://github.com/ECHOES-from-the-Past/mei-analyser/issues/36. For now, let's leave it at the end. I think that having some sort of our "false GABC" (like we discussed here) could solve showing the different distributions among syllables and might suffice:
- [ ] Displaying a highlighted "custom" version of GABC underneath the text (MEI -> "custom" GABC). The highlighting should go on the pitches (or locs) rather than the syllables. It follows the structure of GABC with syl_text(pitches_in_neume/pitches_in_neume), where the / divides the neumes within the syllable, and a regular space ` divides the words. For example:Man(de)du(c)ca(d)ve(fg)runt(afga/ag) et(ga) sa(g)tu(fe)ra(fa)ti(g)` for the beginning of Chant 21.
- [ ] For Aquitanian, display in loc instead of pitches_in_neume. Example: Man(12)du(3)...
- [ ] For Square, use pitches. We are not using the "original" GABC as it encodes absolute positions with a, b, c rather than pitches, which are relative to the clef.

@elsinhadl do you agree with this way of approaching this issue given the many sub-items it has?

elsinhadl commented 4 months ago

I do! Thank you for your excellent work

ECHOES-from-the-Past / mei-analyser