Mitigate wrong encoding under windows

MannLabs / alphabase

Infrastructure of AlphaX ecosystem

https://alphabase.readthedocs.io

Apache License 2.0

31 stars 9 forks source link

Mitigate wrong encoding under windows #182

Closed GeorgWa closed 5 months ago

GeorgWa commented 5 months ago

I'm not a 100% sure about the bug but I think python applies the wrong encoding in some versions of windows or locales. Based on the stack trace it seems like it tries to decode a file with cp1252 encoding and fails doing so.

This change forces utf8 encoding by default which is compatible with ascii.

GeorgWa commented 5 months ago

I think we can remove the errors='ignore' to be on the safe side. Regarding testing, I can't reproduce the bug because the file in #181 itself is a valid utf-8.

jalew188 commented 5 months ago

I think it is the windows issue then, as different Windows in different languages open files in different codes.