eascii: extended ascii in cologne dictionaries

[funderburkjim]

The digitization files xxx.txt (of csl-orig repository) contain many characters besides the standard ascii characters. These are represented in the utf-8 encoding. We use the term 'extended ascii' to refer to any character other than a standard ascii character.

For various reasons, it is useful to survey which extended ascii characters appear in which dictionaries. The 'eascii' directory in this repository aims to provide such survey information.

[funderburkjim]

There are two kinds of summaries: by dictionary and by character.

by dictionary

For each dictionary, show all the extended ascii characters that occur in the dictionary. The eadata directory contains one file for each dictionary. For instance, ea_acc.txt shows all extended ascii characters occurring in the acc dictionary. The lines are ordered alphabetically according to the Unicode code point, and show

• the character
• the unicode code point
• the number of instances of this character in the dictionary
• the name of the unicode character

One detail is that only lines of the file occurring within an entry are considered; excluded are lines representing front matter, appendices, etc. that have not been marked as entries. By convention, an entry begins with a 'metaline' (starts with <L>) and ends with the line <LEND>, and includes all lines between these two lines.

In csl-orig repository, there is an xxx_meta2.txt file for each dictionary, and one component of xxx_meta2 is a listing of the extended ascii characters. For instance, compare ea_acc.txt with acc_meta2.txt. We should strive to have consistency between these two ea lists.

all_ea.txt contains all the individual dictionary files.

[funderburkjim]

by character summary

For some purposes, it is useful to see all the dictionaries which contain a particular character. the 'easummary' files serve this purpose.

There are summary files:

• easummary_greek.tsv for characters whose unicode name starts with 'GREEK'
• easummary_arabic.tsv for characters whose unicode name starts with 'ARABIC'
• easummary_cyrillic.tsv for characters whose unicode name starts with 'CYRILLIC'
• easummary.tsv for the rest of the extended ascii characters occurring in some dictionary.
• easummary_meta.tsv for the extended ascii characters occurring in the metaline of some dictionary.
  • Such occurrences may be erroneous and need to be examined.