Mail filter error when using "special characters"

[Philmo67]

Hello, I encounter errors when subject filters in the email consumption rules contain special characters like an "é":

09/01/2021 12:40 ERROR Rule Compte.Facture Marchand: Error while processing rule: 'ascii' codec can't encode character '\xe9' in position 67: ordinal not in range(128)

09/01/2021 12:40 DEBUG Rule Compte.Facture Marchand: Searching folder with criteria (SINCE 25-Aug-1993 FROM "cgy-serviceclient@marchand.com" SUBJECT "Expédition de votre commande" UNFLAGGED)

09/01/2021 12:40 DEBUG Rule Compte.Facture Marchand: Selecting folder INBOX 

--> if I remove "Expédition" from the filter, everything works fine. It seems that unicode should be used somewhere instead of ascii

[jonaswinkler]

The library defaults to ascii for IMAP query criterias for some reason. I'll look into why that is and use utf-8 instead.

[jonaswinkler]

Alright, without doing too much more digging, this is the current state:

• with 7fef27e, searching with special characters works.
• On Gmail servers, we get an error: imaplib.IMAP4.error: SEARCH command error: BAD [b'Could not parse command']. I can reproduce that.
• We get this error only if special characters are used, according to @sisao (https://github.com/jonaswinkler/paperless-ng/commit/7fef27e6de1acfd28f5f08f0bfc53ad1985cbd36#commitcomment-45873636)
• Works fine on my mail accounts (not Gmail). I can search with special characters now, and that was not possible before.
• Might not work on other servers as well

The fix will stay, since it does not break anything; However, I'm unable to provide any more fixes for this.

[jonaswinkler]

See https://github.com/ikvk/imap_tools/issues/88

[jonaswinkler]

This is definitely an issue of the gmail servers not adhering entirely to the IMAP specifications regarding the SEARCH command.