Add Unicode support

[lemonyte]

Open all files with encoding="utf-8". Fixes #56. Fixes #82.

[Kludex]

Can we use the same technique as @Zac-HD in https://github.com/Zac-HD/shed/commit/a757277a3ca282769fd7a60a8a844c4a87f68d00 ? 🙏

[Zac-HD]

Fwiw Python source code is (these days!) all supposed to be in utf-8; shed does a fancier thing because it also formats code blocks embedded in documentation. But you're welcome to grab the relevant snippet under the local licence rather than AGPL to include it here if you like.

[Kludex]

Fwiw Python source code is (these days!) all supposed to be in utf-8; shed does a fancier thing because it also formats code blocks embedded in documentation. But you're welcome to grab the relevant snippet under the local licence rather than AGPL to include it here if you like.

Ah, ok. Thanks @Zac-HD :)

[Zac-HD]

You may also want to enable https://docs.python.org/3/library/io.html#opt-in-encodingwarning in CI to make sure this doesn't come back. It's a bit fiddly but worth it IMO.

[Kludex]

You may also want to enable docs.python.org/3/library/io.html#opt-in-encodingwarning in CI to make sure this doesn't come back. It's a bit fiddly but worth it IMO.

How the CI will say anything useful about it? I didn't get it 🤔

[Zac-HD]

If you set the environment variable and warnings-as-errors, then your tests will fail if someone adds a new IO call which doesn't specify the encoding even if it would work in the specific cases which you test.