fs_reader: speed up json lines reading

[immerrr]

GzipFile does buffer-related stuff, such as GzipFile.readline in pure Python and does it quite slowly, unlike io.BufferedReader which does it in C.

Check out

• http://spyced.blogspot.com.by/2006/12/wow-gzip-module-kinda-sucks.html
• https://bugs.python.org/issue7471 states that the patch should be applied to py2.7, but it does not, at least for me

[immerrr]

Also, this: https://www.reddit.com/r/Python/comments/2olhrf/fast_gzip_in_python/

[eliasdorneles]

Looks good, thanks @immerrr ! Could you share your benchmarking results?

[immerrr]

The timings are as follows:

$ time python test_gzip.py ds_dump_US_1.jl.gz 
1691290714

real    0m16.099s
user    0m15.968s
sys 0m0.116s

$ time python test_gzip_buf.py ds_dump_US_1.jl.gz 
1691290714

real    0m12.040s
user    0m11.904s
sys 0m0.120s

With test_gzip.py being:

import gzip
import sys

total_bytes = 0
with gzip.open(sys.argv[1], 'rb') as f:
    for l in f:
        total_bytes += len(l)
print(total_bytes)

and test_gzip_buf.py being:

import gzip
import sys
import io

total_bytes = 0
with gzip.open(sys.argv[1], 'rb') as f:
    with io.BufferedReader(f) as bf:
        for l in bf:
            total_bytes += len(l)
print(total_bytes)

[eliasdorneles]

neat, thank you!