Open aarppe opened 4 months ago
Just a note, that some of the entries with a CW_Lemma
do not have matching entry in CW, but are an inflected form of a word in CW, e.g. âcim
, where the definition in CW_Definition
matches âcimêw
instead, which is sufficient to match the entry as a formOf
âcimêw
.
Most of the entries have been merged. Current status is:
+?
) are currently being discarded.3045 new entries from MD
Are there any entries from MD that are not incorporated at all? In my scripting I noticed that there were perhaps just above 300 cases where an MD entry maps to multiple CW entries, which would need manual disambiguation.
3145 AECD entries that have no standardized lemma (
+?
) are currently being discarded.
Concerning these, we'd want to run an old script of mine scrutinizing AECD with the FST again, and see if we could reduce the number of unanalyzed cases (with +?
).
Are there any entries from MD that are not incorporated at all? In my scripting I noticed that there were perhaps just above 300 cases where an MD entry maps to multiple CW entries, which would need manual disambiguation.
For the importjson
currently in itwewina.app, yes. Any entry that does not have a unique mapping to CW (!=1). For the new importjson
, All entries are added, if entries need manual disambiguation they are included as a separate entry. An example of this behaviour change is âhkosiw
, which in itwewina.app only has two CW entries (one with two senses and the other one without). Because there's two, the old behaviour would just give up merging the entries and thus the definitions in MD and AECD would not be merged. New behaviour is to have 4 entries. Eventually we would have a way to force them to correctly merge, but I believe 4 entries to be preferrable to missing entries (especially when someone deselects CW as a source)
Concerning these, we'd want to run an old script of mine scrutinizing AECD with the FST again, and see if we could reduce the number of unanalyzed cases (with +?).
That would definitely be helpful.
I'm also thinking whether we should introduce for each of the dictionaries a persistent unique identifier, that would allow us to link entries unambiguously? This could be based on some existing information, such as the entry head and lexical category, plus then an index to deal with ambiguity - or then it could be just a numeric code. I'd be inclined to consider a transparent PID, but I can be convinced otherwise.
We can definitely add these identifiers to the dictionaries that do not change (MD, AECD, etc.). To account for ambiguities, the index should be recorded in the entries themselves to avoid issues like swapping entries in the source files. We can just add a column on the source TSV files for the identifier, whichever we choose. I do not have particular preferences either way.
I am less certain about introducing identifiers in the CW toolbox files, we could have a discussion about that as well.
How the extensions to the ALTLab version of the Maskwacîs Dictionary were planned, there are two fields that are intended to help establish that an entry in MD can be matched to an entry in CW (at some static point).
CW_lemma
indicates that the MD entry has (at some point) been mapped to an entry in CW. "Lemma" here means "entry head" in the lexicographical sense, rather than "baseform" in the computational sense. There are 7241 such MD entries, cf.CW_lemma
by itself is not sufficient to provide an unambiguous match with a CW entry, so some additional information is needed. Early on, we used the full English definitions as manually copied from CW to the MD database, but since those are under continuous editing, they are not reliable on the long term. See:MD_class
.MD_lemma
(and its associatesMD_stem
andMD_class
) were created to provide the necessary ingredients for including in the LEXC code those MD entries that could not be mapped to CW. There are 2566 such cases.Increasingly, these entries originally missing from CW have yet been added there, so for FST generation purposes our script checks if the combination of the
MD_lemma
andMD_class
map with the\sro
and\ps
fields in CW, in which case they are not added to the LEXC code.Besides the above, there are a number of entries in MD that are neither mapped to CW, nor provided with an
MD_lemma
, etc. While these would not be included in the FST, they could nevertheless yet be included in the *.importjson, but without getting a paradigm.Similar comparisons for LEXC inclusion have not yet been completed in the case of AECD to CW (but not MD).