This allows Python2 pickle files to have strings encoded as either utf8 or ascii (latin1).
The only disadvantage is that it uses the pure-python pickle implementation, rather than the faster C-based _pickle. In practice, though, this doesn't seem much slower.
I thought that I should be able to avoid doing this by setting the errors argument on pickle.load. This gets passed to codecs.decode, and controls what happens if an error happens when trying to decode a string. However, whatever I set as the value of errors, I always get this exception: ValueError: Failed to encode latin1 string when unpickling a Numpy array. pickle.load(a, encoding='latin1') is assumed.
I even tried defining my own error handler with codecs.register_error, but I can't seem to get around that exception.
Anyway, should we go ahead with this? It seems like a bit of a hack, but it does let things work nicely with pickle files that use either encoding.
This allows Python2 pickle files to have strings encoded as either utf8 or ascii (latin1).
The only disadvantage is that it uses the pure-python
pickleimplementation, rather than the faster C-based_pickle. In practice, though, this doesn't seem much slower.I thought that I should be able to avoid doing this by setting the
errorsargument onpickle.load. This gets passed tocodecs.decode, and controls what happens if an error happens when trying to decode a string. However, whatever I set as the value oferrors, I always get this exception:ValueError: Failed to encode latin1 string when unpickling a Numpy array. pickle.load(a, encoding='latin1') is assumed.I even tried defining my own error handler with
codecs.register_error, but I can't seem to get around that exception.Anyway, should we go ahead with this? It seems like a bit of a hack, but it does let things work nicely with pickle files that use either encoding.