Glossary and sign lists

[epageperron]

As part of the lemmatization process and for other tasks, we need to prepare fully consolidated and curated lists to work with. Data must be extracted from different sources, verified, consolidated and prepared in the ideal format to work with. The focus should be on the Ur III data but when we have more data and we prepare an automated extraction system then all data should be gathered to facilitate further research.

• [ ] Extract information from the ePSD
• [ ] Convert Ilya's syllabary to table format
• [ ] Compile lists of information we will gather though annotations (POS, named entity category, semantic category, etc)
• [ ] Prepare a db schema to hold the data from all of our sources in a unified way
• [ ] Extend the DB schema for future LOD work
• [ ] Combine all sources for sign information and verify consistency
• [ ] Combine all sources for vocabulary information and verify consistency
• [ ] Prepare a list of alternating verbal bases (see Thomsen)
• [ ] Prepare extraction scripts so we have data outputs in formats useful for our tools

All relevant files should be in the "pre-processing" folder:

• ORACC master sign list (JSON): https://drive.google.com/file/d/0B8-deXARunnhTFcwX2VFT2VoTGc/view?usp=sharing
• ePSD files (html): https://drive.google.com/open?id=0B8-deXARunnhZno2c1ZLb2Z1UmM
• Ilya contributed a sign list based on his previous work, see here (XML): https://drive.google.com/file/d/0B8-deXARunnhZjJNNFpXTnctdzA/view?usp=sharing
• Lucas prepared a short vs long readings permutation list, see here (table): https://docs.google.com/spreadsheets/d/1u8W7-D_zii8B6B4kolOacitNSubqb-DPYtyIXxMTXfs/edit?usp=sharing
• Prefered readings for the CDLI(table): https://docs.google.com/spreadsheets/d/19r4xgiKS2yM38I6dDTtGRk6QIaM43xywe5BHcPb0LUY/edit?usp=sharing
• CDLI Ur III Master sign list (table): https://docs.google.com/spreadsheets/d/17Tuu5q8pTHQ3e3_W-f3LzNFGluenu-cDWDtWFtKM5tw/edit?usp=sharing
• CDLI glossary (table): https://docs.google.com/spreadsheets/d/1XWLeKlfIIV0okxJTXAAFWTDauLLWCUbKcU_tO9vAVao/edit?usp=sharing
• CDLI sign readings list (table): https://docs.google.com/spreadsheets/d/1744_e2_izpqqI2EgWt6IkBX053se7wLcQOBsl6bJ2dg/edit#gid=0
• Most recent word tokens list ( dating back to June ): https://docs.google.com/spreadsheets/d/13FY2HTQo61QH0bNrVGzcS16gp0tWqmggY1tq2k43Yfw/edit?usp=sharing
• Most recent sign tokens list ( dating back to June ): https://docs.google.com/spreadsheets/d/1zG322-RQbAsrmPoGB3iwOFnQYOo2FEF3F-b6OWVmSAw/edit?usp=sharing Those token lists will soon be updated with @pacinopo's work on the tokenizer.

[epageperron]

Updated to add the list of alternating verbal bases. The file I have is based on the OCR of Thomsen and had been cleaned to get verb - definition so one would have to go over the original file and extract verbs not marked as "regular". Also the ePSD has alternative spellings for words to that might be another source.

[epageperron]

@khoit do you need this for your work? If not, we can maybe postpone to at least after Christmas as a lot of work is left on framrwork? I'd love to work on this