http server file upload support

[QXSoftware]

Hi~

In order to get a http handler that can handle http form post with enctype="multipart/form-data", i.e. file upload, what can I do?

I'm wishing to have a method for instance nng_http_handler_alloc_upload that can deal with an uri, and set an upload directory. It's better to write upload data directly into file system, other than into memory because that could lead to an out-of-memory failure.

[gdamore]

It would be possible to do this but you would need to make your own handler. Right now I have been using post for REST purposes but you could support a POST handler that just did a read with call backs using nng_http_conn_read. To avoid reading the whole thing into memory you would need to use multiple aios of smaller size. Eg read in 8k chunks or something.

Unfortunately there is no convenience function for this. It should not be hard to create one.

On Sat, Oct 13, 2018, 12:00 AM Hunter notifications@github.com wrote:

Hi~

In order to get a http handler that can handle http form post with enctype="multipart/form-data", i.e. file upload, what can I do?

I'm wishing to have a method for instance nng_http_handler_alloc_upload that can deal with an uri, and set an upload directory. It's better to write upload data directly into file system, other than into memory because that could lead to an out-of-memory failure.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nanomsg/nng/issues/750, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPDfVbT6rrt0EsEF7_-bsxZhLj4AAwbks5ukY97gaJpZM4Xafdr .

[QXSoftware]

Thank you for the reply! I think maybe I can do some work to support this need.

[lalanikarim]

Disregard: I missed the nng_http_handler_collect_body call.

@gdamore Is there an upper limit for request body size? I have created a simple web-server using nng_http and I am able to parse the request body for form posts as well as file uploads if the file sizes are small enough. However, when I upload even a moderately large file - 80MB - the request returns back a status 400 - Bad Request. Please advise.

[gdamore]

So is nng_http_handler_collect_body working for you then? Note that using this function for very large posts -- e.g. 80MB, is probably not something you want to do on a public server, because the requests are collected into memory backed by malloc() -- meaning this is going to eat a lot of RAM as the original poster noted.

Making a real upload server out of this is probably not something terrifically difficult, but it hasn't been a real priority for me -- this is pretty far outside the original purpose for the HTTP support in NNG.

[gdamore]

There is an option to adjust the upload limit btw. But yeah, its pretty risky to do that on an internet facing server.

[mikehouse]

Hello. I've set the server. For some GET requests the server gives binary files to the clients. To respond to a client with binary data I'm using nng_http_handler_alloc_file handler. It works great for small files, but fails for big ones, because of out of memory error. I see that underhood there is internal default handler callback http_handle_file https://github.com/nanomsg/nng/blob/7a0de1b25287f08b73c04d4f9c2834ae265cc382/src/supplemental/http/http_server.c#L1393

How can I modify this handler to be able to write synchronously binary data to the connection/response by buffer ? May be there is some example of this ? I see it like this (original code + modifications)

static void http_handle_file(nni_aio *aio)
{
    nni_http_handler *h   = nni_aio_get_input(aio, 1);
    nni_http_conn *   conn= nni_aio_get_input(aio, 2);
    nni_http_res *    res = NULL;
    int               rv;
    http_file *       hf = nni_http_handler_get_data(h);
    const char *      ctype;

    if ((ctype = hf->ctype) == NULL) {
        ctype = "application/octet-stream";
    }

    if ((rv = check_file_access(hf->path) /* do not read file yet */) != 0) {
        uint16_t status;
        switch (rv) {
        case NNG_ENOMEM:
            status = NNG_HTTP_STATUS_INTERNAL_SERVER_ERROR;
            break;
        case NNG_ENOENT:
            status = NNG_HTTP_STATUS_NOT_FOUND;
            break;
        case NNG_EPERM:
            status = NNG_HTTP_STATUS_FORBIDDEN;
            break;
        default:
            status = NNG_HTTP_STATUS_INTERNAL_SERVER_ERROR;
            break;
        }
        if ((rv = nni_http_res_alloc_error(&res, status)) != 0) {
            nni_aio_finish_error(aio, rv);
            return;
        }
        nni_aio_set_output(aio, 0, res);
        nni_aio_finish(aio, 0, 0);
        return;
    }
    if (((rv = nni_http_res_alloc(&res)) != 0) ||
        ((rv = nni_http_res_set_status(res, NNG_HTTP_STATUS_OK)) != 0) ||
        ((rv = nni_http_res_set_header(res, "Content-Type", ctype)) != 0)) {
        nni_http_res_free(res);
        nni_aio_finish_error(aio, rv);
        return;
    }

    // How synchronously write the data to conn/res ?

    size_t buf_size = 8192;
    uint8_t buffer[buf_size];

    FILE *file = fopen(hf->path, "r");
    size_t read;
    nng_iov iov;
    nng_aio *aio2 = NULL;

    nng_aio_alloc(&aio2, NULL, NULL);
//  nng_stream *stream = nng_aio_get_output(aio, 0);
    nng_stream *stream = nng_http_conn_get_stream(conn);
    while ((read = fread(buffer, 1, buf_size, file)) > 0) {
        iov.iov_len = read;
        iov.iov_buf = buffer;
        nng_aio_set_iov(aio2, 1, &iov);
        nng_stream_send(stream, aio2);
        nni_aio_wait(aio2);
    }

    fclose(file);

        nng_aio_stop(aio2);
    nng_aio_free(aio2);

    nni_aio_set_output(aio, 0, res);
    nni_aio_finish(aio, 0, 0);
}

Of course this doesn't work. Thanks a lot.