Correctly closing a stream?

[lithiumlab]

1. What is the proper way of closing a stream? Upon a cancellation by the user i want to terminate and close as gracefully as possible all current requests without closing the connection to be ready for next set of requests to an external API which only allows 10 streams at a time. It seems some of them are getting stuck and pile up.

The application doesn't require some of those requests bodies to be read. Only send and consider OK with a 204 (No content). So i'm only doing .read() on less than 50% of them.

[DEBUG] (Thread-10 ) recv for stream 27 with set([1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, <any-number-here>, 255 ]) already present

Something like this?

    def close_streams():
        if len(connection.streams) > 0:
            for stream in connection.streams:
                connection.streams[stream].close()

Getting this:

    self.connection.streams[stream].close()
RuntimeError: dictionary changed size during iteration

This is what im trying to avoid:

TooManyStreamsError: Max outbound streams is 10, 10 open

Any alternatives or suggestions?

It looks like https://github.com/Lukasa/hyper/issues/264 is related

Another related question is:

2. Is it safe to ignore the Error raised by the intentional cancellation of requests?

This is done in the front-end calling xhr.abort() and then python raises the error in the logs. How should i capture it as gracefully as possible and pass or should i be considering not to?. I'm using Flask in the the frontend. The logs error doesn't seem be directly generated from the app code and i haven't worked enough with sockets and inner parts of the request process to feel confident. Any suggestions on how to approach this?

----------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 721, in sendall
    v = self.send(data[count:])
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 687, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
----------------------------------------

Nothing comes in the ouput with this:

        broken_pipe_exception = IOError
        try:

           stream_id = self.connection.request('POST', url, headers=headers, body=payload)
            resp = self.connection.get_response(stream_id)

        except broken_pipe_exception as exc:
            if broken_pipe_exception == IOError:
                if exc.errno != EPIPE:
                    print '========= Broken Pipe Exception Case 1 ============='                    
                    raise
                else:
                    print '========= Broken Pipe Exception Case 2 ============='

[Lukasa]

You're right that you need to call close(). Your problem is that you're iterating over the streams dict, which gets updated with each call to close(). So what you need to do is this:

stream_ids = list(connection.streams)
for stream_id in stream_ids:
    connection.streams[stream].close()

That way you avoid the issue.

As to EPIPE, you shouldn't ignore it: you should just stop processing once you hit it. EPIPE is raised when the remote peer has closed the connection but we haven't processed it yet, and instead tried to write to the socket. That means there's no point in doing graceful shutdown: the connection is gone anyway. That means that if EPIPE is raised, you should just consider your shutdown process complete (maybe logging that you couldn't finish graceful shutdown because the remote peer got in your face).