Difficult cases for French tokenisation

[bguil]

Here are the cases where I'm not sure about how to tokenise. Should we split or not at the * position? @maxamb: your opinion?

M. + Proper name

• Voici Mademoiselle*Curtis .
• Un docteur a examiné M.*Brown .
• M.*Zhao est mort à Pékin .

Dates

• Le capitaine_Scott a découvert l' île_Scott en décembre*1902 .
• Carl_Smith est mort le 8*août .

Appositions

• Le capitaine*Scott a découvert l' île*Scott en décembre_1902 .
• Senecal habite le Manoir*Cortland .

Il y a

• Yunus a fondé la banque Grameen il*y*a 30 ans .
• Marilyn_Monroe est morte il*y*a 33 ans .

[siyanapavlova]

I would say, if we are to follow the (unofficial) PMB Manual and the Towards Universal Semantic Tagging paper, then the words which should have different semantic tags, should also be separate tokens.

M. + Proper Name

• M./Mademoiselle are considered as titles, so in all three cases, the semantic tags will be ROL PER => two separate tokens

Dates

• we have DOM, MOY, etc. tags, so "décembre 1902" and "8 août" are both split into two tokens (examples in section 4.4 in the unofficial manual)

Appositions

• "capitaine Scott" - again, title, so the semantic tags are ROL PER => two tokens
• I'm not sure, but probably separate tokens, as per section 5.1.2 of the manual. 39/1667 would be an example
• The English version is "Senacal resides in Cortland Manor.". Does that refer to this Cortland Manor (note that it's spelled differently from what is in the dataset)? If yes, then it's a proper name, so it is one token.

This made me wonder if, e.g. Buckingham Palace would also be one token. Probably yes (see 23/1396, where "Great Pyramid" is also one token). But then again, in French, it is palais de Buckingham, which sounds more like "the palace Buckingham / the palace that is named Buckingham" rather than "the building/ that is named Buckingham Palace".

Il y a

I don't know how to treat "il y a", I leave this to the native speakers for now :)

[thebugcreator]

I have created a potato tokeniser for the easy cases. For these difficult cases I'll follow the discussion.

[thebugcreator]

I was looking for a list of French titles to implement and I found this website. Is it useful, or should I look for information elsewhere? Also, what is the limitation in terms of covering the titles for PROPN? I was about to pick the general ones {"Monsieur", "Madame", "Mademoiselle"} and their acronyms {"M.", "Mme", "Mlle"} only.

[bguil]

For the rules which uses lexical information, this information should be considered as a parameter that can be adapted later on.

For French titles, we can start with the list of the website and adapt it later if needed.

[bguil]

I've decided the tokenisation of the remaining cases, following @siyanapavlova's comment. For the last case (il y a), I've choosen:

1. If it used as a frozen expression like in il y a 30 ans [30 years ago], then it is one token il_y_a
2. If it used as a translation of there is, then it is "compositional" and contains 3 tokens: il y a (note that in this case, we can have negation (il n'y a pas) or another tense (il y avait).

In the 163 sentences, we have 2 examples for both cases:

Yunus a fondé la banque Grameen il_y_a 30 ans .
Il y a aussi des touristes français .
Il y a un biscuit sous la table .
Marilyn_Monroe est morte il_y_a 33 ans .