search

Thriftpy / thriftpy2

Pure python approach of Apache Thrift.
MIT License
572 stars 91 forks source link

Python 3.7: RuntimeError: read() called while another coroutine is already waiting for incoming data #169

Open klairetan opened 3 years ago

klairetan commented 3 years ago

I'm running into basically the same issue as https://github.com/encode/httpx/issues/527. When running client requests concurrently, the request ends up throwing with a stack trace like below

packages/thriftpy2/contrib/aio/client.py", line 42, in _req return (yield from self._recv(_api))
File "/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/client.py", line 56, in _recv fname, mtype, rseqid = yield from self._iprot.read_message_begin()
File "/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/protocol/binary.py", line 251, in read_message_begin api, ttype, seqid = yield from read_message_begin( 
  File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/protocol/binary.py", line 28, in read_message_begin sz = unpack_i32((yield from inbuf.read(4))) 
   File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/transport/base.py", line 47, in read return (yield from readall(self._read, sz)) 
    File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/transport/base.py", line 15, in readall chunk = yield from read_fn(sz - have) 
     File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/transport/buffered.py", line 40, in _read buf = yield from self._trans.read(max(rest_len, self._buf_size)) 
      File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/transport/framed.py", line 41, in read yield from self.read_frame() 
       File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/transport/framed.py", line 46, in read_frame buff = yield from readall(self._trans.read, 4) 
        File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/transport/base.py", line 15, in readall chunk = yield from read_fn(sz - have) 
         File"/var/lib/conda/envs/ktan/lib/python3.8/site-packages/thriftpy2/contrib/aio/socket.py", line 169, in read buff = yield from asyncio.wait_for( 
          File"/var/lib/conda/envs/ktan/lib/python3.8/asyncio/tasks.py", line 483, in wait_for return fut.result() 
           File"/var/lib/conda/envs/ktan/lib/python3.8/asyncio/streams.py", line 684, in read await self._wait_for_data('read') 
            File"/var/lib/conda/envs/ktan/lib/python3.8/asyncio/streams.py", line 503, in _wait_for_data raise RuntimeError( RuntimeError: read() called while another coroutine is already waiting for incoming data

This tends to happen when I run two client methods concurrently. For example, when I run the following snippet 1000 times, it would throw the above error around 40% of the time.


bar, foo, zed = await asyncio.gather(
   thrift_client.get_bar(...),
   thrift_client.get_foo(...),
   thrift_client.get_zed(...)
)

Since all requests share the same StreamReader, I wonder if when we make requests concurrently, it's possible we end up calling self._wait_for_data('read') from a second request that also tries to run another read coroutine? Has anyone run into this before? The folks working on httpx got around it by adding a lock around trying to read from the stream.

ethe commented 3 years ago

There is not a reporting before, I will try to reproduce your case, thank you