Check for issues with language coding

[AaronGullickson]

I need to do a deeper dive into the difference between language and languaged to make sure I am capturing the relevant categorization whenever possible.

[AaronGullickson]

Ok, I have looked through this and the general codes are definitely not good enough. They provide a lot of detail for European languages but then lump vast geographic regions together (e.g. "Sub-Saharan African").

The detailed codes work well in many cases, although the 1980 data often provide too much detail because nothing was re-coded from what respondents provided. Ideally, I would also want to look at a linguistic measure of language similarity but this would be quite an undertaking.

I think what I will do is follow a procedure of using the detailed codes with adjustments made to make the two time period more comparable. Specific changes as follows.

[AaronGullickson]

In general, I will use the detailed codes with the following adjustments:

• All languages below 2500 (European) will be collapsed to the 100s place (e..g 810 becomes 800), except for the following:
  • 110 (Jamaican Creole) will be kept separate from English
  • 420 Afrikaans will be kept separate from Dutch
  • 1140 (Haitian Creole) and 1150 (Cajun) will be kept separate from French
  • 2310 (Croatian) and 2320 (Serbian) are kept separate.
• Hindi and Urdu (3101, 3102, 3103, 3104) are collapsed into 3100 because that is the only category in 1980.
• All codes under 35XX are collapsed into 3500 (Uralic).
• All codes under 37XX are collapsed into 3700 (Other Altaic).
• All codes under 43XX are collapsed into 4300 (Chinese). Mandarin and Cantonese codes do not seem consistent across time period and many just identify as Chinese speakers.
• All codes under 47XX are collapsed into 4700 (Thai/Siamese/Lao). Thai and Laotian are not separable in the 1980 codes.
• Collapse all 52XX codes into 5200 (Indonesian) except 5270 (Malay).
• Collapse all 53XX codes into 5300 (Other Malay).
• Collapse 58XX codes into 5800 (Near East Arabic Dialect).
• Big problem with Native American languages which were all collapsed in 1980. For consistency I will also collapse them in the later data.

There is also the problem of groupings of "other" or "nec" languages. These should all probably be given one code and not be considered endogamous with anything else. These would be the following codes:

• 3140 and 3150 (Indian and Pakistani, nec)
• 3190 (other Indo-European languages)
• 5290 (Other Asian Languages)
• 6200 (Other Afro-Asiatic Languages)
• 6390 and 6400 (Africa, nec)
• >=9300 , all the other NEC codes