mode='t' support for gzip, bz2 & lzma

[maxgrenderjones]

The read-compressed-file modules all added a 't' mode for opening files in text mode at the same time 't' was added as a mode for plain vanilla open. Is it possible to monkey-patch them so that they work too?

P.S. sadly the answer isn't as simple as wrapping them in a TextIOWrapper as they only partially support the file protocol:

>>> sys.version
'2.7.9 (default, Apr  7 2015, 07:58:25) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]'
>>> with gzip.open('foo.gz', 'r') as gz:
...     with io.TextIOWrapper(gz, encoding='utf8') as f:
...             for line in f:
...                     print(f)
... 
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
io.UnsupportedOperation: read1

(same issue on python 3.2 - works fine on 3.3)

One workaround that wfm (found at https://bugs.python.org/issue12591) is to alias read to read1 like so:

>>> with gzip.open('passwd.gz', 'r') as gz:
...     gz.read1=gz.read
...     with io.TextIOWrapper(gz, encoding='utf8') as f:
...         for line in f: 
...             print(line)

[maxgrenderjones]

Having spent a while trying to fix this, there are a series of challenges:

• In 2.6, the things you get back from gzip.open / bz2.BZ2File are missing most of the methods that TextIOWrapper depends on, so it's not a simple as copying over read1
• In 2.7 the gzip version of the workaround above works, but bz2.BZ2File is a C-like thing (not a pythonic object), so you can't assign arbitrary attributes
• In <3.3, gzip/bz2 don't have an open method (and the File versions are still missing methods such as read1 which are required to get TextIOWrapper to work), and if you don't call open in 3.3+ with t then newline behaviour isn't what you want

One option would be to use external packages that work around this issue:

• bz2file (claims to provide a modern API-compatible BZ2File and open)
• backports.lzma (a backport of the 3.3+ lzma module which therefore has a modern API, even on earlier pythons, but requires a C module, you'd probably only want to make use of it if it was already available)
• ?? (haven't found an equivalent for gzip)

[kislyuk]

Thanks for looking into it! My philosophy is actually to only support 2.7 and 3.3+. I'll play around with a possible set of wrappers today, but the challenges you found are good to know.