search

eidheim / Simple-Web-Server

A very simple, fast, multithreaded, platform independent HTTP and HTTPS server and client library implemented using C++11 and Boost.Asio. Created to be an easy way to make REST resources available from C++ applications.
MIT License
2.62k stars 758 forks source link

Server truncates requests if they are sent in chunked transfer #173

Closed sharpie7 closed 6 years ago

sharpie7 commented 6 years ago

Back in #29 it was commented: Q) Does the server support handling chunked transfer? A) No chunked transfer would have to be implemented by you on top of Simple-Web-Server...

In fact, there is a problem which means that the server in the library doesn't properly pass the contents of chunked HTTP requests in the request parameter of the callback routine. Currently, I don't think that an application can implement support for chunked transfer in requests to the server unless there are some changes to the library.

In a chunked transfer there is no content-length header. Looking at server_http.hpp around line 462 we see that if there is no content-length header the code assumes that there is no more data in the body of the HTTP request except for that which has already been read while decoding the headers.

The effect of this is that if a request is sent to the server in chunked mode the callback routine only gets in request->content the characters from the body which were read as part of the header. Characters after that are not available to the callback routine (and I assume are discarded).

Thanks as always for a great library. Hope you will be able to help address this.

eidheim commented 6 years ago

You are right, I did not consider that requests content could be chunked transfer encoded. Have only ever seen response content being chunked transfer encoded. Supporting reading chunked transfer encoded request content, however, would be similar to what has been done in the ClientBase class. For testing purposes, do you know any client that sends its request's content chunked transfer encoded?

sharpie7 commented 6 years ago

Thanks for the fast response. Yes, I think the work done for the client is rather similar.

The project where I found this problem is this one: https://github.com/IoTKETI/Mobius

Unfortunately it requires a bit of effort to install and then configure to get it sending requests. Here is a Wireshark capture of an packet. I guess you could replay this using wget or similar.

I am happy to help by either: 1) Giving you instructions on how to install the tooling I am using, or 2) Test on my machine which is already setup with all the bits.

The second option should be easier!

eidheim commented 6 years ago

Lets go with option 2. I did a quick implementation in the branch chunked_request (https://github.com/eidheim/Simple-Web-Server/commit/9b24f60b3210043e226f60ebcc8abea1edb87592). Let me know how your tests go:)

eidheim commented 6 years ago

I created some tests and it seems to work, let me know if you have any more issues. I changed the above commit a bit through rebase, but the changes has been pushed to master.

sharpie7 commented 6 years ago

Thanks for the amazing response again!

I did a very quick check this morning and it is working fine for my application now. I will code-up some more extensive tests later and double-check it works in the most important scenarios.

sharpie7 commented 6 years ago

Tried with more tests for both valid and invalid chunked encodings. All is working as expected.

I think we're done on this now. Thanks.

eidheim commented 6 years ago

Thank you for testing these changes further!