some bytes in the file are incompatible with the system locale
In this case, fallback to decoding as UTF-8 as a last resort (rather than crashing on the UnicodeDecodeError). This behaviour was added when parsing the request file, rather than in auto_decode as it didn't seem to belong in a generic util (though that util looks to only be ever called when parsing requirements files anyway).
Perhaps we should just go straight to UTF-8 without querying the system locale (unless there is a PEP-263 style comment), per the docs[1]:
Requirements files are utf-8 encoding by default
But to avoid a breaking change just warn if decoding with this locale fails then fallback to UTF-8
For the case where:
In this case, fallback to decoding as UTF-8 as a last resort (rather than crashing on the
UnicodeDecodeError). This behaviour was added when parsing the request file, rather than inauto_decodeas it didn't seem to belong in a generic util (though that util looks to only be ever called when parsing requirements files anyway).Perhaps we should just go straight to UTF-8 without querying the system locale (unless there is a PEP-263 style comment), per the docs[1]:
But to avoid a breaking change just warn if decoding with this locale fails then fallback to UTF-8
[1] https://pip.pypa.io/en/stable/reference/requirements-file-format/#encoding
Fixes: #12771