search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

Thrift timeout when submitting topology #341

Closed tdhopper closed 7 years ago

tdhopper commented 7 years ago

This might not be a bug, but it's new to us. We've been seeing the following on some of our deploys.

I believe that the topology sometimes starts despite this. Other times, it all works fine if we run it again.

Any idea why this might be happening suddenly? Can we increase the timeout time on Thrift? 

Submitting topo topology to nimbus...Traceback (most recent call last):
  File "/Users/w/anaconda/envs/topologies/bin/sparse", line 11, in <module>
    sys.exit(main())
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/streamparse/cli/sparse.py", line 56, in main
    args.func(args)
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/streamparse/cli/submit.py", line 256, in main
    requirements_paths=args.requirements)
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/streamparse/cli/submit.py", line 219, in submit_topology
    env_config, nimbus_client, options=options)
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/streamparse/cli/submit.py", line 97, in _submit_topology
    topology=topology_class.thrift_topology)
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/thriftpy/thrift.py", line 198, in _req
    return self._recv(_api)
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/thriftpy/thrift.py", line 210, in _recv
    fname, mtype, rseqid = self._iprot.read_message_begin()
  File "thriftpy/protocol/cybin/cybin.pyx", line 429, in cybin.TCyBinaryProtocol.read_message_begin (thriftpy/protocol/cybin/cybin.c:6325)
  File "thriftpy/protocol/cybin/cybin.pyx", line 60, in cybin.read_i32 (thriftpy/protocol/cybin/cybin.c:1546)
  File "thriftpy/transport/framed/cyframed.pyx", line 48, in thriftpy.transport.framed.cyframed.TCyFramedTransport.c_read (thriftpy/transport/framed/cyframed.c:1526)
  File "thriftpy/transport/framed/cyframed.pyx", line 68, in thriftpy.transport.framed.cyframed.TCyFramedTransport.read_frame (thriftpy/transport/framed/cyframed.c:1716)
  File "thriftpy/transport/framed/cyframed.pyx", line 31, in thriftpy.transport.framed.cyframed.TCyFramedTransport.read_trans (thriftpy/transport/framed/cyframed.c:1254)
  File "thriftpy/transport/cybase.pyx", line 61, in thriftpy.transport.cybase.TCyBuffer.read_trans (thriftpy/transport/cybase.c:1472)
  File "/Users/w/anaconda/envs/topologies/lib/python2.7/site-packages/thriftpy/transport/socket.py", line 108, in read
    buff = self.sock.recv(sz)
socket.timeout: timed out

cc. @gallamine

dan-blanchard commented 7 years ago

@tdhopper I've encountered this before on flakey wifi. Usually happens when I'm trying to deploy from a coffee shop.