Closed abotsis closed 5 years ago
Howdy, I'm seeing high cpu iterating over a ResponseReader object in Python 3.7.
ResponseReader
When doing something like:
results = service.jobs.export('search *') for _ in results: pass
CPU spikes to 100%, and it's slow. cProfile indicates (trimmed):
cProfile
ncalls tottime percall cumtime percall filename:lineno(function) 1 0.013 0.013 174.869 174.869 <string>:1(<module>) 26612304 11.754 0.000 89.386 0.000 binding.py:1285(peek) 53224608 35.328 0.000 146.927 0.000 binding.py:1304(read) 53224608 59.029 0.000 107.549 0.000 client.py:436(read) 53224607 27.661 0.000 43.139 0.000 client.py:468(readinto)
A little digging reveals two potential performance issues with the ResultsReader.read() method:
ResultsReader.read()
self._buffer
Additionally, adding a few print statements to read() immediately before return shows this pattern:
read()
return
DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0 DEBUG read: 1 size: 1 DEBUG read: 1 size: 0
In the example above, if I instead iterate over results._response, things work as expected. I have not tried to reproduce in Python 2.
results._response
As a side note, while I'm not familiar with the implementation of this in it's entirety, I've noticed:
def read(self, size = None): r = self._buffer self._buffer = b'' if size is not None: size -= len(r) r = r + self._response.read(size) return r
if size=0 (which it does sometimes, apparently) and self._buffer==0, this will pass -1 to the self._response.read call.
-1
self._response.read
Seems like a dupe of #223 - there's some recommended workarounds there. Please re-open if this is different
Howdy, I'm seeing high cpu iterating over a
ResponseReader
object in Python 3.7.When doing something like:
CPU spikes to 100%, and it's slow.
cProfile
indicates (trimmed):A little digging reveals two potential performance issues with the
ResultsReader.read()
method:self._buffer
Additionally, adding a few print statements to
read()
immediately beforereturn
shows this pattern:In the example above, if I instead iterate over
results._response
, things work as expected. I have not tried to reproduce in Python 2.As a side note, while I'm not familiar with the implementation of this in it's entirety, I've noticed:
if size=0 (which it does sometimes, apparently) and self._buffer==0, this will pass
-1
to theself._response.read
call.