asio_service_http not handling long response

[maciek-slon]

I've just created C++ example for speech_to_text service (see post_builder branch f78efef). Service is called properly on platform side, but on the API side there is a bug with handling long responses, sent in multipile TCP packets. Class asio_service_http treats every packet independently, which results in error:

no double return after header

when second part of data is retrieved. As a result for request to say "test" in english "en", first packet received by asio_service_http is:

Content-Length: 39940
Connection: close
Content-type: application/json; charset=UTF-8
Server: Hop

{"payload":"UklGRsR0AABXQVZFZm10IBAAAAABAAEAgD4AAAB9AAACABAAZGF0YaB0AAD//wAAAAD//wAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAP////8AAAEAAAD/////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD//wAAAQAAAAAAAQAAAAAA//8AAAAA/////wAA
AAABAAAAAQD//wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD//wAAAAAAAP////8AAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAD//wAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAEA//8AAAAAAAAAAAAAAAA

and then next packet comes, obviously without the header (as it is just continuation of previous data):

AAAAAAAAAAAAAAAABAAAA/////wEAAAABAAAAAAAAAAAAAAD//wAAAAD//wEAAAAAAP////8AAAAAAAAAAA
EAAQAAAAAAAAAAAAAAAAD//wAAAAABAP//AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAP//AQAAAAAA//8AAAAAAAAAAAEA
AAD//wAAAAAAAAAAAAABAP//AAAAAAAAAAABAAAAAAAAAAAAAAABAAAAAAD//wAAAAAAAAAA/////wAA/////wAAAAAAAP//AAABAAAAAA
D//wEA//8AAAAAAAAAAP//AAAAAAAAAQAAAAEAAAAAAAAAAQAAAAEAAAD//wAA//8AAAAAAQAAAP//AAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAEAAQAAAAAAAAAAAAAAAAD//wEAAAAAAAAAAAABAAEAAQAAAAAAAAAAAP//AAAAAAAAAAD//wAAAAD//wAAAAD//wAAAQD//w
EAAQA

which triggers before-mentioned exception.

[alexge233]

I'm not following. What do you mean by "multiple TCP packets" there's no such thing in HTTP, are you referring to "chunked" transfers?

The error you're getting probably means that a header wasn't detected.

Where is the HTTP header being sent from the platform back to the API? The current asio implementation will ignore data without a header. The HTTP header you have posted does not have the Transfer-Encoding: chunked.

My guess is the response header is either malformed, or non-existant. Have you tried without chunked responses?

[maciek-slon]

It is not my service, but speech_detection_sphinx4. And, as far as I understand, HOP is responsible for chunking. Last part, which service developer is responsible of, is sending JSON response as a whole

sendResponse( hop.HTTPResponseJson(response) );

(line 217, https://github.com/rapp-project/rapp-platform/blob/devel/rapp_web_services/services/speech_detection_sphinx4/svc.js)

[alexge233]

That is bizarre I tested it months ago with multiple files. My guess is @klpanagi has changed something since then. Regardless, this header response does not seem to be correct.

[klpanagi]

Every Platform Web Service returns an application/json response as a part of the Platform's communication server specifications.

Though, that is not the issue you investigated there...

Check Platform's active branch. As we informed you a few days ago, the cpp-api works only for the api_stable branch of the RAPP-Platform.

If this error still occures, please report back

EDIT On api-stable branch: I executed the speech_recog c++ example and i could not reproduce the error.

[alexge233]

UPDATE I've seen this behavior recently whilst implementing TLS/SSL handlers:

it happens when the HTTP endpoint (server) uses Transfer-Encoding: chunked which is not implemented and handled by the C++ API.

HOP/HTTP service must provide non-chunked responses (200) with a valid content length. I am willing to bet this is the reason, and the solution is on the HOP-side, and not in the API.

The HTTP Header you've posted indicates its not chunked, but please verify by printing the entire header.

[maciek-slon]

Data in the first post is complete - there is nothing more received by the client.

[alexge233]

@maciek-slon I've replicated the issue.

The response in asio_service_http is complete, for example if you try:

std::cout << json << std::endl;

Put this line after line 208.

Right after copying the response buffer, you will see the entire message. I tested this with chunked response.

With non-chunked response (using Content-Length) I kept getting an exception of "no double return after header" implying that the asio callback was finishing too early in line 215.

Removing the callback (comment it out) I got the full response to display. So the problem is a bug in the logic of the callback chaining!

Thank you for reporting this, this could have creeped silently and become a major issue! I will fix this ASAP.

[alexge233]

@maciek-slon Bug fixed. Please note that the WAV base64 encoded string appears to be invalid. You can see the pushed changes in branch ortelio especially in class asio_service_http.hpp. However bear in mind I'm still working on this branch, so it will change again.

Also note that we do not process chunked responses at the moment!