MichaelFenwick
commented
2 months ago Some thoughts on how to handle this:
It's too hard to identify apostrophes directly, letting anything that remains be considered a single quote. Beyond plural possessives (the boys' game) looking like the end of a single quoted sequence, apostrophes are also used for indicating accents (stormin', ol') in a way that isn't readily identifiable, and which can't reliably be expected to match a dictionary entry. As such, unambiguous punctuation needs to be removed from the sentence, and then some logic can be used to predict whether what remains is an apostrophe or closing single quote (CSQ).
- Start by identifying anything where the
[’']character is between two letters and exclude that as potential closing single quote (PCSQ). - Also check for
[’']characters which precede the first'(opening single quote (OSQ) character). If OSQs are used, then anything before the first one is an apostrophe and not a CSQ. - Once simple contractions are excluded, check for the existence of two or more
[’']characters in the sentence. If only one is found, that must also be a contraction, and thus an apostrophe. - If multiple PCSQs remain then:
- Check to see if they can be unambiguously paired up with OSQs (that is, no two PCSQs occur in a row without a preceding OSQ). For those where this is the case, assign them as a CSQ and remove from the list. For this step, words which start with an apostrophe can be assumed to be an OSQ (it's possible for this to be an apostrophe in a partially pronounced word instead ('em, 'n', '80s) but this is going to be more rare than OSQ I believe, and most of these cases can probably be identified by checking a static list gathered from some corpus).
- It's worth noting that with leading apostrophes, that these might be written with either an apostrophe, OSQ, or CSQ character. If it's the latter, we can convert and treat it as an apostrophe, however.
- For the ones that remain, for each set of consecutive PCSQs, identify any at the end of a word when preceded by an
s(likely plural possessive) orin(likely contraction ofing). If all but one in the set fits this, consider that one a CSQ, and the others apostrophes. - Potentially before the previous step, or maybe at the same time, or maybe after, check to see if any of the PCSQs appear to end a "thing"
- If the PCSQ is attached to a word that starts with an apostrophe or OSQ, consider it a CSQ (note that
'n'is a notable exception to this). - If the PCSQ is adjacent to other punctuation (either preceding, such as a period or comma, or following, such as a double closing quote), it is likely a CSQ.
- If the PCSQ precedes the end of the string, then it is likely a CSQ.
- If the PCSQ is attached to a word that starts with an apostrophe or OSQ, consider it a CSQ (note that
- Check to see if they can be unambiguously paired up with OSQs (that is, no two PCSQs occur in a row without a preceding OSQ). For those where this is the case, assign them as a CSQ and remove from the list. For this step, words which start with an apostrophe can be assumed to be an OSQ (it's possible for this to be an apostrophe in a partially pronounced word instead ('em, 'n', '80s) but this is going to be more rare than OSQ I believe, and most of these cases can probably be identified by checking a static list gathered from some corpus).
Single quotes need to be identified and isolated from apostrophes. They can be replaced with the appropriate Unicode single quote characters, while apostrophes can remain the apostrophe character (or changed from closing single quote to apostrophe if it started as one).