Documentation of lemmatization decisions in English corpora

[nschneid]

We should write a summary if this does not already exist, since https://universaldependencies.org/u/overview/morphology.html#lemmas says 'treebanks have considerable leeway in interpreting what “canonical or base form” means'.

For example, e9884f8 removed plural endings from lemmas in proper names: knight~s~ inn and kc chief~s~. Was this intentional? (@jheinecke)

It also removed plural endings from pluralized ethnonyms like Caucasians, which was intentional. These are also tagged PROPN.

I know other edge cases with lemmas arise from time to time.

[nschneid]

It seems EWT does not lowercase capitalized forms for proper noun lemmas. Another question is whether lowercase forms that are conventionally capitalized should have capital lemmas, and if so whether this counts as a typo or abbreviation.

Other issues:

113 - numbers

99 - quotation marks

69 - abbreviations

[amir-zeldes]

FYI in GUM we also singularize, but don't decapitalize NNPS lemmas, a practice I think we took over from the behavior of TreeTagger in the distant past.

On some level, I think it makes sense that the lemma of "Investigations" in "Federal Bureau of Investigations" is "Investigation", since it's a compositional, transparent plural, and for other NNPS that are opaque names we generally want to keep caps (e.g. "Sony" is the lemma of "Sony"). So taking these two desirables together (if we agree they're desirable), would seem to lead to the practice you're describing, right?

[nschneid]

Spelling out details of capitalization policies:

• Decapitalize or keep capitalization in PROPN lemmas? Currently EWT and GUM preserve capitalization.
• Decapitalize or keep capitalization in other NNP lemmas (e.g., ADJ modifiers within a name)? Some discussion in #164, resolution TBD.
• Decapitalize or keep capitalization in proper ADJ lemmas (e.g., Jewish)? ~Currently these are lowercased.~ Lemmas retain capitalization in EWT in #144.
• Add capitals or keep lowercase in lemmas for PROPNs (and even proper ADJs) that are conventionally capitalized? And if so, should the stylistic choice to lowercase be registered in a feature? ~Currently anything lowercase in the input form just stays lowercase in the lemma.~ Capitalized in EWT: months and days in #142, demonym-type proper nouns/adjs in #144. Others TBD—many lowercase PROPNs remain lowercase in lemmas but should probably be capitalized. No feature commenting on capitalization.
• Convert nonstandard ALL CAPS words to title case in names? GUM does this, EWT may do this in the future.
• Capitalize lemmas for other types of conventionally capitalized words, e.g. the pronoun "I"? ~Currently these are kept as in the input in EWT.~ Pronoun "I" is capitalized in EWT in #142.
  • Alphabetic numbers (Roman numerals, or A, B, C etc. in a list) would be another interesting case. The capitalization may matter in a certain numbering scheme but they have the same numeric interpretations.

Regarding inflectional morphology in lemmas within proper names:

• Some names are compositional, which is an argument for removing the plural in the lemma. Investigations and Securities are transparently plural even in organization names. Knights Inn refers to a medieval-themed hotel, though it does not involve actual knights.
• The policy could be to lemmatize nouns in names as if they were common nouns, but preserving capitalization that arises from proper name status.
• This seems reasonable as we analyze internal morphosyntax even within names. E.g. for Tyson's Corner (location) we would analyze the possessive clitic regardless of MWE status. Thus, plurals should arguably be removed in lemmas regardless of MWE status.

Lemmas of pronouns:

Documented at https://universaldependencies.org/en/pos/PRON.html

• In EWT, personal pronouns are lemmatized in the nominative (them => they), and ~attributive possessive pronouns in the nominative (our => we)~ possessive forms are normalized to the attributive (theirs => their) (#293). Number is not normalized.
• ~As noted above, in EWT, the capitalization of pronoun I or i is left as in the input.~ With #142, the lemma of pronoun I or i is capitalized.
• Demonstrative plural pronouns (these, those) are normalized to singular in the lemma.
• Reflexive pronouns (e.g. ourselves) ~and predicative possessives (theirs)~ are not normalized.

Lemmas of numbers:

• In EWT, thousands separator commas are omitted from lemmas with #142.
• Pluralized numerals:
  • If it is a regular plural, like AK-47's, the plural ending should not be part of the lemma. If the plural form has a special meaning, like 1990s meaning the decade starting with 1990 (this pattern is available for multiples of 10 generally), the plural ending IS part of the lemma: #467
  • Apostrophes included as part of the plural of a number should be omitted from the lemma: #335.
• other issues: #113

Lemmas and spelling variation:

• A general principle could be that if the lemma differs from the wordform in any respect other than capitalization, it should be explained in features.
  • if the spelling seems to be accidental: Typo=Yes, CorrectForm=(normalized spelling)
  • abbreviated months and days of the week—Oct, Tues.: Abbr=Yes (#143)
  • shortenings for economy, whether it would appear in edited text or not—yrs, luv, thx, prepositional 2: Abbr=Yes
  • gon (na), wan (na), c'm (on): Abbr=Yes, lemma=go/want/come
  • colloquial shortened pronunciation—eatin: Abbr=Yes, lemma=eat, CorrectForm=eating? (raised in #144)
  • expressive spelling choices beyond capitalization (#68) are now Style=Expr
  • other dialectal spelling variation (color/colour, labeled/labelled): ???
    • EWT does not currently normalize favor/favour, behavior/behaviour

PS the actual expansion of FBI is singular Investigation, but that's not really relevant here. :)

[amir-zeldes]

Oh, thanks, I learned something there :)

Note that some of these behaviors are not shared with GUM:

• Conventional capitalization is restored in lemmas, for example lemma of "i" is "I": https://corpling.uis.georgetown.edu/annis/#_q=ImkiIF89XyBsZW1tYT0iSSI&_c=R1VN&cl=5&cr=5&s=0&l=10
• This extends to non-standard lower case NNP, for example sony > Sony: https://corpling.uis.georgetown.edu/annis/#_q=InNvbnkiIF89XyBsZW1tYQ&_c=R1VN&cl=5&cr=5&s=0&l=10
• possessives are lemmatized separately - our > our: https://corpling.uis.georgetown.edu/annis/#_q=Im91ciIgXz1fIGxlbW1h&_c=R1VN&cl=5&cr=5&s=0&l=10

[arademaker]

some inconsistencies in the lemmatization of PRON?

ar@leme ud-english-ewt % awk '$4 ~ /PRON/ {print $2,$3,$4}' *.conllu | sort | uniq -c | sort -nr
...
 390 our we PRON
   4 ours ours PRON
...
 758 your you PRON
   1 your your PRON

[kanayamah]

@arademaker mine, yours and theirs are kept as they are, so should ours.

[AngledLuffa]

Any decision on capitalizing nationalities etc used as an adjective? American, for example. I could put that together as a pull request too

[nschneid]

If we have a scalable way to distinguish proper adjectives from other adjectives, IMO it would be good to treat them similarly to proper nouns, which would preserve the initial capital if it is present in the wordform. So this would update the second bullet in my list above.

AFAIK the lemmas do not actively capitalize proper nouns that have been spelled without capitalization.

[AngledLuffa]

I think it just needs to be done once for ewt, maybe once for some of the other treebanks. Anyway, the scaling would be by making a long list of country names and grepping for them.

On Fri, Apr 9, 2021, 11:19 AM Nathan Schneider @.***> wrote:

If we have a scalable way to distinguish proper adjectives from other adjectives, IMO it would be good to treat them similarly to proper nouns, which would preserve the initial capital if it is present in the wordform. So this would update the second bullet in my list above.

AFAIK the lemmas do not actively capitalize proper nouns that have been spelled without capitalization.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/UniversalDependencies/UD_English-EWT/issues/131#issuecomment-816869811, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2AYWMODWQMTFEZPYYC77LTH5AJNANCNFSM4YFSXQZA .

[amir-zeldes]

In GUM, lower case proper names are upper cased in the lemma if the name is usually spelled capitalized (example: sony -> lemma=Sony). Nationalities are also capitalized in adjective lemmas.

To find a good list of these, you can also check all NORP entities in the enamex layer of OntoNotes, which mostly correspond to demonym adjectives.

[AngledLuffa]

African-American - capital, capital? African ringneck - capital, lowercase? Alexandrine parrot - ??? Indian capital Delhi - capital, lowercase, capital? Kolkata is an Indian state - capital, ..., capital, lowercase? and if it's misspelled kolkatta, it gets turned into Kolkata? the only West Indian spot - capital, capital? or is West lowercase? a huge West Indian population - same question, I guess American palette - capital, lowercase? authentic Jamaican food - lowercase, capital, lowercase?

On Fri, Apr 9, 2021 at 11:55 AM Amir Zeldes @.***> wrote:

In GUM, lower case proper names are upper cased in the lemma if the name is usually spelled capitalized (example: sony -> lemma=Sony). Nationalities are also capitalized in adjective lemmas.

To find a good list of these, you can also check all NORP entities in the enamex layer of OntoNotes, which mostly correspond to demonym adjectives.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/UniversalDependencies/UD_English-EWT/issues/131#issuecomment-816891295, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2AYWONELASW4CLLU5DZK3TH5ERBANCNFSM4YFSXQZA .

[nschneid]

Those all sound good to me. I would include terms like West (Indian) or Mr. (Smith) as part of the name and therefore capitalized.

[AngledLuffa]

Not entirely sure why I took this on. Must be a chip on my shoulder for being the Nth EWT annotator out of N. It's pretty tedious though.

More possible ambiguities: Aryan, Masonic - capitalized? air asia -> Air Asia southeast asian -> Southeast Asian central asian republics -> Central Asian republic irish coffee -> Irish coffee canan t3i (a product name, misspelled) U$ -> US Panamal Canal -> Panama Canal (is currently Panamal Canal) Suez canal -> Suez Canal Soul food? weirdly capitalized as Soul Food, but lemma should still be "soul food" christmas? Goes to Christmas?

[AngledLuffa]

also, satanic & satanism?

[amir-zeldes]

These all look good to me. FWIW Wikipedia capitalizes Satanism and Satanic: https://en.wikipedia.org/wiki/Satanism

Here's another one for discussion: what should we do about "eldest":

• Pierre Charles became the eldest son

It's tagged JJS and looks pretty compositional, but obviously there is also "oldest"...

[nschneid]

eldest: lemma=old, OldFashioned=Yes, Human=Yes ;)

[amir-zeldes]

OK, going with old. This all makes lemmatization a much more interesting layer to predict/use as a feature!

[AngledLuffa]

There is a word "elder", though, which I wouldn't consider a form of "old". "The village elder". Maybe it should also be "old", though

[nschneid]

• the elder child: comparative adjective, lemma = old
• the village elder: noun

[AngledLuffa]

Running into some alternate word forms and abbreviations

christmas -> Christmas easter -> Easter halloween -> Halloween christ as a swear word... Christ? anti-american -> anti-American ? latin - american -> Latin - American ?

Antichrist or antichrist? company name such as American Pride Irrigation & Landscaping - all capitals? u.k goes to...? rep.ireland ...?

Calif abbreviation kept as Calif ? publication title such as The American Conservative ... The and Conservative?

[nschneid]

For general vocabulary used in names (titles, companies, etc.), the common/proper distinction is tricky. @amir-zeldes what does GUM do for lemma capitalization in such names?

[nschneid]

My gut feeling is that capitalization in the lemma shouldn't be too context-specific—so "the" should always have a lowercase lemma even in names. But I don't know what the simplest thing is in practice.

[AngledLuffa]

A bunch more possible edge cases in country names:

french fries ... French? german shepherd ... German shepherd? Indian ringneck ... Indian? Guinea pig? Iranian Navy? capital N or not? Islamism? Islamist? phila pa ... Phila PA? norweigan cruise line -> Norweigan royal carribean -> Royal Carribean ? chicken parmesan -> chicken parmesan ? neoplatonic Shakespearian Siamese lynx spartan conditions pad thai

[AngledLuffa]

The resolution of some of these questions aside, I changed a whole bunch of them in this pull request:

https://github.com/UniversalDependencies/UD_English-EWT/pull/144

[nschneid]

You could check a dictionary or style guide for some of these. I would lowercase "guinea pig" but in most cases I share your intuitions.

[amir-zeldes]

For general vocabulary used in names (titles, companies, etc.), the common/proper distinction is tricky. @amir-zeldes what does GUM do for lemma capitalization in such names?

The real question is a tagging question IMO: If we go with NNP/PROPN then if the conventional spelling of that name/brand/whatever is capitalized, the lemma will be capitalized as well (and singularized if NPS).

You've inspired me to do some consistency checks on GUM, so I'm seeing we weren't always 100% consistent on this, but that was the intention, and I should be able to fix a bunch of cases.

[amir-zeldes]

@AngledLuffa I would do everything the same as you except:

• Fix Norweigan to Norwegian in lemma
• phila -> Philadelphia (probably with Typo=Yes)

Note also for spartan, dictionary.com concurs the non-demonym usage is lower-cased, so I suppose it's just a lexicalized adjective and no longer supposed to be referencing Sparta.

Oh, and yes, "elder" can be a noun, also based on the existence of the plural "elders", so that's a separate lemma

[AngledLuffa]

Oops, completely missed the correct spelling of Norwegian. Which brings up another question: Norwegian Forest Cat. Capitalize Norwegian Forest, no forest, or also Cat?

[nschneid]

• phila -> Philadelphia (probably with Typo=Yes)

Abbr=Yes, I think. Probably not accidentally shorter.

Oops, completely missed the correct spelling of Norwegian. Which brings up another question: Norwegian Forest Cat. Capitalize Norwegian Forest, no forest, or also Cat?

I'd lean toward the conservative side and just capitalize Norwegian. (Similarly: Labrador retriever, etc.)

[AngledLuffa]

Slight disagree based on the wikipedia article, but I don't feel strongly:

https://en.wikipedia.org/wiki/Norwegian_Forest_cat

[amir-zeldes]

The Wiki convinced me, I think it's [[Norwegian Forest] cat] (a cat from the the Norwegian Forest), rather than a Norwegian variety of [forest cat], so Norwegian Forest could be a place name?

[amir-zeldes]

publication title such as The American Conservative ... The and Conservative?

Forgot to say about "The" & co - the PTB guidelines tag function words in names normally, so the lemmatization practice derived from that has been to lemmatize them normally. For example:

The/DT/the Lord/NNP/Lord of/IN/of the/DT/the Rings/NNPS/Ring

This is what we do in GUM in any case - only the NNP-tagged words receive capitalized lemmas as relevant.