HTTPlib not correctly parsing headers

[arnoutdekimo]

Hi,

I am running the assistnow example, and trying to use the httplib to parse the result of offline data. This -sometimes- works, but sometimes the API returns length 0, and sometimes it even returns wrong data.

(EDIT -- this is wrong -- see posts below ) _When further looking into the library code, and sniffing the UART bus, I noticed the following:

• In cellCallbackfirst the status code is attempted to be parsed. This is done in cellFileResponseReadStatusCode
• The amount of data read over the bus is hardcoded set at some 64 bytes (U_HTTP_CLIENT_CELL_FILE_READ_FIRST_LINE_LENGTH)
• This is then attempted to be parsed, and expects to see the response code. This succeeds
• A bit later, the total response size is attempted to be parsed from the headers. This is done by reading U_HTTP_CLIENT_CELL_FILE_READ_HEADERS_LENGTH bytes. The previous read out data is discarded though

This is an incorrect way of parsing the data though. Note in what the first 64 bytes look like in my case:

The content-length header is already in there, and is cut in half. Meaning the subsequent parsing trying to find this length fails.

The correct way of parsing, is to read out chunks, and -bundle these-, and only discard the amount of data that is already parsed. In this case the bug is that in the first 64bytes useful data remained that was lost. It should have been appended to the subsequent data until all headers were found.

Depending on how fast data comes in over the bus, encapsulation methods (CMUX or not), how big the headers are, etc, this code might and might not have worked._

Kind regards, Arnout

[arnoutdekimo]

I seem to be wrong - the code reads from a file, at a given offset. Hence why it is able to read the same data multiple times..

[arnoutdekimo]

Analysis attempt #2

In the code that reads headers, it reads 1024 bytes from the file at the right offset:

It then returns 411 in that function

Below in the function, it (in this case) sets the correct length

And indeed, this time the code executes correctly. And every time I step through the code with the debugger, it seems to work.

When I don't hit breakpoints.. it does not work anymore, and things get corrupted.

=> Will look into the timing issue, but so never mind the first analysis..

[RobMeades]

You're not having an easy time with this are you? Sorry! Let me know as soon as I can usefully help.

[arnoutdekimo]

Man, ive lost time with this.

Stuff I did:

• Futher ensure there was no HW issue on UART or something that could lead to this

• Extensively tried to follow the code, and dump buffers at various places in the code, and compare with good/bad scenario's. But the code is so complex + every time you run the example data gets ever-so-slightly changed that its really hard to compare

• I even disabled my DMA acceleration in favor of simple 1-char-per-interrupt behavior

• .. And it still randomly worked and sometimes it didn't

BUT, I seem to have found the trigger : the CMUX.

I'll make a new issue for this, and close this one, since othertwise this might get too confusing

[RobMeades]

Interesting; we do test running over CMUX, both with just the GNSS-inside-cellular case and more generically "CMUX used all of the time"; obviously not enough.