Cannot send emails with non-ascii characters

[sentry-io[bot]]

There are actually four issues in Sentry that this affects. I'll link them all to this one, which has the most instances. We need to fix this quickly because it's affecting our alerts.

I'm not sure what caused this, but you can easily reproduce it by using the link here:

https://www.courtlistener.com/reset-password/

The error started about 20 hours ago, when I merged the token counting PR:

https://github.com/freelawproject/courtlistener/pull/3963/files

It came with a new dependency for tiktoken, but no other dependency updates as I understand the poetry.lock file to work.

So:

• Why did this start happening?
• Does it affect others (if so, can we fix it upstream)?
• How do we fix it as quickly as possible?
• How do we get the upstream fix in place (assuming we make one/there is one)?

Sentry Issue: COURTLISTENER-71K

UnicodeEncodeError: 'utf-8' codec can't encode characters in position 48-50: surrogates not allowed
(13 additional frame(s) were not displayed)
...
  File "cl/alerts/tasks.py", line 348, in send_alert_and_webhook
    connection.send_messages(messages)
  File "cl/lib/email_backends.py", line 226, in send_messages
    email.send()

[sentry-io[bot]]

Sentry Issue: COURTLISTENER-71M

[sentry-io[bot]]

Sentry Issue: COURTLISTENER-71P

[sentry-io[bot]]

Sentry Issue: COURTLISTENER-71V

[mlissner]

This is affecting a lot of the Django community. @ERosendo when you have a moment, will you share what you tried and what worked, since I know you tried a bunch of things? It is probably also worth it to link to the Django PR and issue so that others can find this too.

[sentry-io[bot]]

It looks like this is affecting bots.law too. I think we just need to pin the same version of Django over there, too, right?

Sentry Issue: BIGCASES2-3H

[ERosendo]

It looks like this is affecting bots.law too. I think we just need to pin the same version of Django over there, too, right?

You're right. We implemented the fix in https://github.com/freelawproject/bigcases2/pull/519 and haven't had any related errors in Sentry since then.