UAlbertaALTLab / crk-db

Managing the Plains Cree dictionary database
https://itwewina.altlab.app/
GNU General Public License v3.0
0 stars 2 forks source link

Mapping MD to CW #125

Open aarppe opened 4 months ago

aarppe commented 4 months ago

How the extensions to the ALTLab version of the Maskwacîs Dictionary were planned, there are two fields that are intended to help establish that an entry in MD can be matched to an entry in CW (at some static point).

  1. CW_lemma indicates that the MD entry has (at some point) been mapped to an entry in CW. "Lemma" here means "entry head" in the lexicographical sense, rather than "baseform" in the computational sense. There are 7241 such MD entries, cf.
cat crk/dicts/Maskwacis_altlab.tsv | gawk -F"\t" 'NR>=2 { if($12!="") print; }' | wc -l
    7241
cat crk/dicts/Maskwacis_altlab.tsv | gawk -F"\t" 'BEGIN { while((getline < "crk/dicts/Wolvengrey.tsv")!=0) { gsub("ý","y",$1); entry[$1]++; } } { if($12 in entry) print entry[$12]; }' | sort | uniq -c
4282 1
 144 2
  12 3
   1 4
  1. MD_lemma (and its associates MD_stem and MD_class) were created to provide the necessary ingredients for including in the LEXC code those MD entries that could not be mapped to CW. There are 2566 such cases.
cat crk/dicts/Maskwacis_altlab.tsv | gawk -F"\t" 'NR>=2 { if($16!="") print; }' | wc -l
    2565
cat crk/dicts/Maskwacis_altlab.tsv | gawk -F"\t" 'NR>=2 { if($12=="" && $16=="") print; }' | wc -l
     140

Similar comparisons for LEXC inclusion have not yet been completed in the case of AECD to CW (but not MD).

fbanados commented 4 months ago

Just a note, that some of the entries with a CW_Lemma do not have matching entry in CW, but are an inflected form of a word in CW, e.g. âcim, where the definition in CW_Definition matches âcimêw instead, which is sufficient to match the entry as a formOf âcimêw.

fbanados commented 4 months ago

Most of the entries have been merged. Current status is:

fbanados commented 4 months ago

Some extra work could be done to merge entries.
Screenshot 2024-07-24 at 5 45 55 PM

Screenshot 2024-07-24 at 5 56 40 PM

Screenshot 2024-07-24 at 5 57 14 PM

aarppe commented 4 months ago

3045 new entries from MD

Are there any entries from MD that are not incorporated at all? In my scripting I noticed that there were perhaps just above 300 cases where an MD entry maps to multiple CW entries, which would need manual disambiguation.

3145 AECD entries that have no standardized lemma (+?) are currently being discarded.

Concerning these, we'd want to run an old script of mine scrutinizing AECD with the FST again, and see if we could reduce the number of unanalyzed cases (with +?).

fbanados commented 4 months ago

Are there any entries from MD that are not incorporated at all? In my scripting I noticed that there were perhaps just above 300 cases where an MD entry maps to multiple CW entries, which would need manual disambiguation.

For the importjson currently in itwewina.app, yes. Any entry that does not have a unique mapping to CW (!=1). For the new importjson, All entries are added, if entries need manual disambiguation they are included as a separate entry. An example of this behaviour change is âhkosiw, which in itwewina.app only has two CW entries (one with two senses and the other one without). Because there's two, the old behaviour would just give up merging the entries and thus the definitions in MD and AECD would not be merged. New behaviour is to have 4 entries. Eventually we would have a way to force them to correctly merge, but I believe 4 entries to be preferrable to missing entries (especially when someone deselects CW as a source)

Concerning these, we'd want to run an old script of mine scrutinizing AECD with the FST again, and see if we could reduce the number of unanalyzed cases (with +?).

That would definitely be helpful.

aarppe commented 3 months ago

I'm also thinking whether we should introduce for each of the dictionaries a persistent unique identifier, that would allow us to link entries unambiguously? This could be based on some existing information, such as the entry head and lexical category, plus then an index to deal with ambiguity - or then it could be just a numeric code. I'd be inclined to consider a transparent PID, but I can be convinced otherwise.

fbanados commented 3 months ago

We can definitely add these identifiers to the dictionaries that do not change (MD, AECD, etc.). To account for ambiguities, the index should be recorded in the entries themselves to avoid issues like swapping entries in the source files. We can just add a column on the source TSV files for the identifier, whichever we choose. I do not have particular preferences either way.

I am less certain about introducing identifiers in the CW toolbox files, we could have a discussion about that as well.