Closed vtermanis closed 5 years ago
Read the exact requested character count from file-like objects read()
method. This results in significantly more calls to said method (as e.g. type markers result in a one byte read) and leads to considerable performance degradation.
If file-like object is seekable()
(e.g. file on local filesystem), enable buffering, otherwise use no buffering (e.g. socket).
coverage_test.sh
with compiled extensionpympler
reference leak test (see bottom of test.py
)python3 -mubjson fromjson test/data/CouchDB4k.compact.json /tmp/CouchDB4k.compact.ubjson
python -mtimeit --n 50000 \
-s "
from io import BytesIO
from ubjson import load, __version__
print(__version__)
raw = BytesIO()
with open('/tmp/CouchDB4k.compact.ubjson', 'rb') as f:
raw.write(f.read())
"\
'raw.seek(0); load(raw)'
Before
0.13.0
0.13.0
0.13.0
50000 loops, best of 3: 23.4 usec per loop
After
0.14.0
0.14.0
0.14.0
50000 loops, best of 3: 73.8 usec per loop
Before
0.13.0
0.13.0
0.13.0
50000 loops, best of 3: 25.5 usec per loop
After
0.14.0
0.14.0
0.14.0
50000 loops, best of 3: 92.5 usec per loop
python3 -mtimeit -r1 -n1 -s "
from ubjson import load, __version__
print(__version__)
" "
with open('DEFRA.uk_air.ubj', 'rb') as f:
load(f, intern_object_keys=True)
"
Before
0.13.0
1 loops, best of 1: 2.08 sec per loop
After
0.14.0
1 loops, best of 1: 5.53 sec per loop
seek()
-able file-like object (as before)
Problem
When decoding from a file-like object (using
load()
via C extension), up to BUFFER_FP_SIZE bytes are buffered whenread()
ing from it. This means that data is likely to be consumed past the end of the encoded ubjson block which is subsequently not available to either decode additional ubjson blocks or for other purposes.Potential solutions
seek()
back to recover unused read data.seek()
Test case
Expected result
All 10 documents are decoded.
Actual result