All the code in redun.File and friends assumes we have a single valid utf-8 string for the path.
But python accepts bytes as the path objects too. This is needed when we're working with filesystems that encode filenames using something else than UTF-8.
There's some functions that crash when trying to give File a bytes path:
File: get_filesystem_class() - get_proto() - the urlparse method fails on non-utf8 byte strings
Dir: all of the above, and also concatenating the glob pattern - complains that TypeError: Can't mix strings and bytes in path components
All the code in
redun.File
and friends assumes we have a single valid utf-8 string for the path.But python accepts
bytes
as the path objects too. This is needed when we're working with filesystems that encode filenames using something else than UTF-8.There's some functions that crash when trying to give
File
abytes
path:get_filesystem_class()
-get_proto()
- the urlparse method fails onnon-utf8
byte stringsTypeError: Can't mix strings and bytes in path components
The workaround is to hack:
I also tried changing the self.classes.File but it can't be overwritten (uses getitem) - so one would have to replace this whole FileClasses thing.
I think one of two things can be done here:
FileClasses
,get_filesystem_class
and friends, without so much monkeypatchingpathlib.Path
as requested in https://github.com/insitro/redun/issues/8get_proto()
andurlparse
would still crash when given non-utf8 bytestringsWhat do you think?