search

pistacheio / pistache

A high-performance REST toolkit written in C++
https://pistacheio.github.io/pistache/
Apache License 2.0
3.16k stars 695 forks source link

File uploading? #75

Open MattMacGillivray opened 7 years ago

MattMacGillivray commented 7 years ago

Is file uploading supported?

kennymalac commented 7 years ago

I am interested in this as well, I saw that there is a FileBuffer class and I was wondering if I could see a usage example or something similar. At the moment I'm unsure how to output an octet-stream and receive multipart JSON/file data

kennymalac commented 7 years ago

Actually I found http://pistache.io/guide/#static-file-serving so it looks like that is documented but I'll have to dig around the source code to find out the specifics of how to handle multipart buffer data.

kennymalac commented 7 years ago

Pistache doesn't support Content-Disposition or Content-Transfer-Encoding headers required for multipart/form-data so submitting a file in a form is not possible. I'll see what I can do to get some code working for these headers... though my C++ is rusty (not the programming language lol)

https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

kennymalac commented 7 years ago

After a bit of research it appears most HTTP servers ignore Content-transfer-encoding, even nodejs's http-server...

EDIT: taking a look at https://github.com/pillarjs/multiparty/blob/master/index.js

kennymalac commented 7 years ago

I've done a little progress on this so I'm wondering if @oktal can provide feedback once I get it semi-complete... I will try to submit a prototype for multipart data & file parsing as soon as I can as it's a huge requirement for a project I'm working on and I really like this REST framework!

kennymalac commented 7 years ago

Unfortunately I'm going to need some help with this. There are multiple issues and I'm finding it hard to work around Pistache's C-style custom buffer classes. I have my WIP here: https://github.com/kennymalac/pistache/commit/824322ae05d41800ad8e38eae83fb3a96e16df16

There are several issues. One is that I need a DynamicStreamBuffer that can output to a temp file, if a file is too large it will crash the program because the Heap won't have enough space for a large file. so I wanted to use the Chunking mechanism, but there is a problem with that... The chunking is tied to the logic for HTTP chunking, but I have to detect multitype before the Transfer-Encoding parser apply() method is called. The reason is that we don't want to inititialize say, a 8GB buffer, only to find out later that it's not even encoded correctly. So I want to do "partial pre-parsing" where I encode the form data in some sort of FormData() class. The current parser is setup to just output a single buffer, with a single cursor per Request body parse, so I'm thinking that this will be a problem, because the cursor advance function continues to the end of the request body, rather than doing boundary checking, which is needed for multipart requests. So I have to delay the creation of the message->body_ buffer because there are limits to what can be stored in the parsed body response. What I'd like is that the returned body has access to tmpfiles somehow... Browsers support Transport-Encoding alongside multipart and the way I have it layed out that is not possible. I'd like to make chunking work with multipart data but I don't know the best way to do it.

As for the Content-Disposition header, I haven't done that. I ran into a problem with MultipartContentType because I need the mixin behavior from ContentType so the code doesn't use the MultipartContentType header. I realized after writing that that the http_header.cc file only has HTTP Header headers, not Headers that can be parsed in the HTTP body.

If I could get some suggestions or feedback that would be great. I would continue on this but I'm a bit lost on how to move forward. Also yes I know my indention is bad (2spaces).. :-P (I'll git reset any formatting errors and refmt later)

bocse commented 5 years ago

Is there a workaround available for uploading moderate-sized files (5-10MB) with reasonable speed (without it taking one second), without any writing to disk or chunking? (i.e. assuming there is enough RAM available)

While Pistache is very useful and versatile (especially in combination with OpenAPI v3), this issue renders it almost unusable in a production setting.

Looking forward to any suggestion.

Thank you.

dennisjenkins75 commented 5 years ago

Since this bug was filed the internal receive buffer mechanism was updated.

Have you ran a benchmark? Suggestion: Create a minimal pistcahe server that allocates a 20M receive buffer, handles "POST" to "/" and then does no processing. Then test it using a client (wget, curl, python w/ requests, etc...)

Are your results repeatable?

What are your system specs?

Can you publish your benchmarking tools for peer-review?