CU-8695ucw9b deid transformers fix

[mart-r]

Since transformers==4.42.0, the tokenizer being loaded is expected to have the split_special_tokens attribute. However, the version we've saved (and load) won't have that attribute. So the processing fails (an exceptions is raised). This failure (along with the exception) is logged. But the overall process is never halted.

What this PR does:

• Creates a workaround for the underlying issue
  • Adds the split_special_tokens attribute to the tokenizer if required
• Creates a way for these failures to be more easily noticed and fixed in the future
  • By limiting the TransformersNER class to only a pre-defined (10 by default) number of similar consecutive excptions
  • If more than the specified number of consecutive similar exceptions are caught, the last one is raised instead
  • This way, if this happens again, after the 10th (by default) consecutive issue, the process is halted and the issue can be seen and (hopefully) fixed

[tomolopolis]

Task linked: CU-8695ucw9b Fix issue with DeID models and transformers 4.42+

[mart-r]

why not just raise the exception and halt the process?

My best guess is that at some point in the past, something in the try-except block was raising an exception due to something that was unable to be fixed (perhaps some kind of undefined characters that caused spacy to fail? I don't know) and thus all the exceptions were being caught to allow the rest of the pipeline to still keep working.

The reason I left in the massive try-except block is because there's potentially users that rely on it.

Though in principle, I agree. The exception should be raised since otherwise (most likely) the DeID pipe will most likely simply not do anything. And personal information will be revealed where it shouldn't.

I will do the following:

• Look into running some DeID models without the try-except block on various data
• See if/what part has the possibility of raising an exception (and what kind) that we can recover from

And then I'll report back

• Either we let the exception be raised as it is
• Or we handle the few that I find in a reasonable way (where possible)

[mart-r]

Went through the DeID of a document that had some characters in the middle of the target text that could potentially cause issues.

• Tried
  • Control sequences (\x00 and \x1F)
  • Long character seqences (a, ` and\n` repetated 100 000 times)
  • Some random special characters (§, ¶, ¿)

And the result was:

• Everything was successfully de-identified on either side of the added 'special' part was just as well with or without the additional text

As such, I don't really see any reason for the try-except to exist and I removed it. So next time we support a transformers version that doesn't work with the saved model, we'll see the crash - and reason for it - immediately.