Finishing up Key-Pano database...

[fractaldragonflies]

Update to this issue added Mar 13, 2023 in reply to my earlier note and subsequent reply from Mattis. See further below.

The current form was based on IDS with our orthographic profiles used to calculate segmented BIPA and LingPy cognate detection methods used to calculate CogId, CogIds, and BorId. We've since edited the database using EDICTOR, revising BorIds, adding source (known donor language and form), add '!PB! annotation to Note field.

Need now is to put database into its final form for use and distribution. In some way we need to replace the original forms.csv calculated from IDS with a new forms.csv that includes the fields we add and revised as well as some complete added entries. And remove the original creation step so that it cannot be mistakenly overwritten with just IDS entries.

So this would seem to be a new generation of the Key-Pano with new name too -- to avoid confusion of databases?

=== Some issues to resolve ===

• Source
  • Not often not labeled when donor is other than Spanish or Portuguese
    • Label it “Unknown” in those cases?
    • Many cases of this.
  • Separate source into known donor language and known donor form?
• BorId other than blank or 0 indicates cross-family borrowing
  • BorId by itself does not indicate which of the languages with same BorId is source (if any).
    • OK. That is not its purpose.
  • As long as source labeled (above) I guess OK.
  • It just feels a bit messy to me!
  • Remove ‘0’ values since no different than ‘ ‘ value.
• Family/sub-family
  • No distinction between Panoan and Tacanan families - all given as Pano-Tacanan.
  • Is this OK?

[LinguList]

Luckily, this is not the first time to do it. And we have examples from the SEABOR workflow. I suggest I take care of feeding the data back in and will keep @fractaldragonflies in the loop on all steps. What is good is that we can "clean" what is not that clean in EDICTOR now, and will also check for consistency.

[fractaldragonflies]

Returning to issue of finishing up our KeyPano database.

Here I repeat some of my commentary of my Mar 6 email, with some of response from @LinguList and a bit further processing for my part. Maybe with references to other issues as needed.

(1) It seems the next steps would be to a) clean up the database as needed still, and then b) model it again to the cldf standard (as our starting keypano database).

(2) @LinguList has agreed to look at modeling the annotations in cldf, and will work with me to teach me how to do it.

Observations

I did a brief review of the current database, making a spreadsheet from the EDICTOR format of the 'forms' relation:

• entries = 23,272
• SOURCE = 1,050 [source annotated borrowings which includes donor_language and donor_value]
• LOAN == TRUE from IDS 778 [this may not be needed in final keypano]
• BORID = 1,958
• 461 from Spanish and Portuguese entries [these are not targets of our study]
• 1,497 effect borrowing cognates remain from Peruvian and neighboring languages
• 447 BorIds do not have a corresponding source specified.
• NOTE == !PB! = 198

Cleanup and cldf modeling

• Create ‘Donor_language’, ‘Donor_value’ from ’Source' [I've already done this in a script.]
• Annotate ‘Donor_meaning’ for the 1050 source annotated borrowings. [@LinguList suggests this could be practical.]
• Keep ‘LOAN’ from IDS. [Not essential, but keeping it let's us compare borrowing annotations.]
• Drop SOURCE once we've modeled Donor_language, Donor_value.
• Change NOTE from text to Partial_borrowing = TRUE, FALSE field. [I've already done this in a script, taking care to be sure that the word is already treated as a borrowing.]
• Simplify language names (doculect names): PortugueseBR => Portuguese, SpanishLA => Spanish. [Yes they were coded as LA and BR varieties, but I don't envision ever having multiple varieties of Spanish or Portuguese at the same time.]

ISSUES

I'm less sure what to do in these cases:

• For BORID without a corresponding Source, I have set Donor_language to 'Unknown' in my script. [This thus treats all entries with BORID as borrowings. Donor_value is left empty.]
• But it's probably that case that some of these shared BORIDs indeed are donors instead of recipient cases. [Maybe in some cases, Roberto could determine the source without extensive study?]
• Sub-family is not specified. Should we include this? [Probably not too much work in the cldf languages relation, where we need only specify once per language.]
• We could drop: alignment, autocogid, autocogids, cogid, cogids, autoborid. [They were part of @LinguList automatic determination of BORID, but maybe not relevant given the updated BORIDs. However, the alignment might be informative?]

[fractaldragonflies]

@fractaldragonflies recalled that I had added several entries to the Spanish donor the key-pano database. Specifically, I had add 34 entries to the IDS dictionary. The .pdf attached to this note documents those new entries. I've since marked the NOTE field as !NEW! for these new entries. Just as !PB! can be coded as a boolean indicator of partial borrowing, the !NEW! can be coded as a boolean indicator of addition to the original IDS Spanish data.

key-pano-db-sp-adds.pdf

[fractaldragonflies]

Quechua is the largest donor after Spanish. If we include a Quechua wordlist, this would provide more opportunity for the multi-lingual methods to detect borrowings from Quechua.

WOLD includes Imbabura Quechua (Ecuador region). While not from the Quechua subgroup closest to the Pano-Tacanan languages, it might still be an adequate representative of the entire Quechua family for multilingual sequence comparison methods.

Should we add Imbabura Quechua to the Key-Pano database?

[LinguList]

Yes, this sounds like a good idea. It will delay our process of annotation, but I could try to give it some priority. It would also require me fiddling around with EDICTOR settings. But it may maybe really be worth it.

The easiest way to proceed is to create a manual Quechua wordlist with concept glosses as we use in EDICTOR / Key-Pano (thus, wold glosses, I assume) and segment it maybe manually already. This wordlist, presented as TSV file to me, can then by me be included as part of EDICTOR.

[LinguList]

Annotation would be probably faster this time, since one can filter EDICTOR entries for Quechua words and then compare them.

[fractaldragonflies]

Annotation would be probably faster this time, since one can filter EDICTOR entries for Quechua words and then compare them.

Will work on this today to get Quechua wordlist as tsv file.

[LinguList]

Nice!