Incomplete Response in HTTP1.1

[deshmukhrajvardhan]

UPDATE: i printed the self._buffer_end value and in the failed cases it is always = 65536 I assumed that this value should be <=65535.

Should the new_buffer() be called before this happens?

OLD MESSAGE: Hi Lucas, I had a problem with HTTP1.1 library. (After making 90 serial 'GET' method calls) The Wireshark pcap trace shows the full response but the BUFFER over here shows incomplete response.

This self._sock.buffer and the recv_into in self._sock.fill() is provided by socket library, Right?

Any tips on how to debug this?

[deshmukhrajvardhan]

Problem Identified: Overview:

1. In the (common to http1 and http2) buffsocket.py file a new buffer (space allocation) is created if the current one is filled before it is used by the other files

2. In the case of the other some files, the data is read before the buffer in buffsocket.py is full, so this case is never encountered.

Problem:

1. In other file cases, the buffsocket.py buffer after the second last file has only 71 bytes left, BUT as the condition [1] is: "Only if 0 space remaining in the buffer create a new one (allocate more space)" The 22nd segment (size>71 Bytes) starts to download the buffer is full only 71 Bytes of the response of next file are stored.
2. This stored value is given to the parser which sees incomplete response and says Response=None [2] which fails in here [3], and causes error and TCP RST message to be sent.

Soultion: I have set this [4] code:

     if self._remaining_capacity<=11935:

self.new_buffer()

and it seems to work.

Help: Does it sound reasonable the keep the above comparison value a constant?

Thanks, Raj

Links:

[1] https://github.com/Lukasa/hyper/blob/75f25f73c17a16cb1f2ca7d46f390b0dfe4925dc/hyper/common/bufsocket.py#L165 [2] https://github.com/deshmukhrajvardhan/hyper/blob/ecf17b6bc143d3af75757bca43b1794000a96ae2/hyper/http11/parser.py#L75 [3] https://github.com/deshmukhrajvardhan/hyper/blob/ecf17b6bc143d3af75757bca43b1794000a96ae2/hyper/http11/connection.py#L291 [4] https://github.com/Lukasa/hyper/blob/75f25f73c17a16cb1f2ca7d46f390b0dfe4925dc/hyper/common/bufsocket.py#L164