Able to send multipart/form-data

[daurnimator]

As requested by @fur-q

2016-01-27 13:29:47 daurnimator furq: so the other day I played with multipart form encoding 2016-01-27 13:29:53 daurnimator furq: I'm wondering how you envision the API 2016-01-27 13:31:49 furq er 2016-01-27 13:32:19 furq i guess the same as a normal post but allowing for file handles as values 2016-01-27 13:33:31 furq i take it the api for a form-urlencoded post is request(url, { key = "value" }) 2016-01-27 13:33:34 daurnimator furq: so this is where I got up to: https://github.com/daurnimator/lua-http/compare/WIP-multipart-form-data?expand=1 2016-01-27 13:33:34 furq or something similar 2016-01-27 13:34:04 daurnimator furq: the thing about those is that you can provide all sort of fun things in the parts: they all have their own headers section 2016-01-27 13:34:32 daurnimator a section usually has a least content-type. but also often content-disposition 2016-01-27 13:34:40 furq oh yeah 2016-01-27 13:34:58 furq it's been ages since i've touched any of this 2016-01-27 13:35:32 daurnimator furq: so the current encoding function I've got there, you provide an iterator that returns headers, body pairs 2016-01-27 13:36:00 furq it would be nice if you could just do a post with a file handle and have it automatically generate multipart with the file set to application/octet-stream 2016-01-27 13:36:09 daurnimator where body can be a string, a file object, or another iterator... 2016-01-27 13:36:19 furq as far as manual control goes you probably know better than i do 2016-01-27 13:36:58 daurnimator (because multipart/form-data sometimes contain another multipart/form-data as one of their parts.... its often very nested) 2016-01-27 13:37:28 furq i can see why so many http libraries don't bother with this 2016-01-27 13:38:23 furq also isn't the content-disposition always form-data 2016-01-27 13:38:54 daurnimator furq: no. it can be e.g. attachment 2016-01-27 13:38:56 furq oh never mind 2016-01-27 13:39:20 furq for forms it's always form-data but it can also be 'form-data; name="foo"' or whatever 2016-01-27 13:39:38 daurnimator furq: except for file uploads. 2016-01-27 13:40:01 furq that's not what this w3 page says 2016-01-27 13:40:17 daurnimator in which case it can be attachment; name="foo"; filename=somethingascii; filename*=someunicodeencodingmess 2016-01-27 13:40:38 furq maybe the rfc says different 2016-01-27 13:41:06 furq actually the rfc doesn't mention that either 2016-01-27 13:41:31 daurnimator furq: there's a whole IANA registry for it 2016-01-27 13:41:38 furq nice 2016-01-27 13:41:41 daurnimator furq: https://www.iana.org/assignments/cont-disp/cont-disp.xhtml 2016-01-27 13:42:22 furq "attachment" says it's for emails 2016-01-27 13:43:35 daurnimator furq: splitting hairs here; but nothing says that you can't serve an email over HTTP. 2016-01-27 13:43:39 zash You can do attachment in HTTP too, I think browsers ask you were to save it then 2016-01-27 13:43:42 furq "multipart/form-data" contains a series of parts. Each part is expected to contain a content-disposition header [RFC 2183] where the disposition type is "form-data", and where the disposition contains an (additional) parameter of "name", where the value of that parameter is the original field name in the form. 2016-01-27 13:43:43 daurnimator furq: perhaps you're confusing HTML vs HTTP? 2016-01-27 13:43:45 daurnimator zash: yep. 2016-01-27 13:44:21 furq i'm only interested in multipart/form-data 2016-01-27 13:44:33 daurnimator furq: e.g. https://support.microsoft.com/en-us/kb/260519 2016-01-27 13:44:37 zash Uploading files uses some disposition in a form-data thing too 2016-01-27 13:44:40 furq i wasn't aware any other multipart http requests existed, but you can always count on http to have some fucking dismal hole 2016-01-27 13:45:11 furq er 2016-01-27 13:45:18 furq isn't that for responses? 2016-01-27 13:45:26 daurnimator furq: well that link is. 2016-01-27 13:45:31 TheCycoONE a bunch of multiparts 2016-01-27 13:45:43 furq that has nothing to do with multipart though 2016-01-27 13:45:48 daurnimator furq: but why does that matter? this code path is hit for both client and server. 2016-01-27 13:46:02 zash https://en.wikipedia.org/wiki/MIME#Mixed-Replace is fun btw 2016-01-27 13:46:45 zash Server push from 1998 :) 2016-01-27 13:46:49 furq oh christ, multipart responses? 2016-01-27 13:46:55 furq that sounds really awful 2016-01-27 13:47:01 zash furq: No it's awesome 2016-01-27 13:47:17 zash Especially the mixed-replace one, which means you can animate ASCII and stuff 2016-01-27 13:47:36 TheCycoONE saves a lot of tcp sessions 2016-01-27 13:47:51 daurnimator furq: so at this point; I've very tempted to say "get a MIME library; not lua-http's problem" 2016-01-27 13:48:14 furq form-data posts are pretty common though 2016-01-27 13:48:15 zash daurnimator: luasocket includes one... 2016-01-27 13:48:25 furq i don't care if you ignore all the other edge cases that seemed like a good idea in 1998 2016-01-27 13:48:27 daurnimator zash: luasocket also includes an SMTP library. 2016-01-27 13:48:56 zash and base64 encoding 2016-01-27 13:49:08 TheCycoONE ... why not a luacurl that wraps everything curl supports? 2016-01-27 13:49:10 furq apparently browsers don't even support multipart/mixed etc any more 2016-01-27 13:49:18 furq at least firefox dropped it ages ago 2016-01-27 13:49:22 daurnimator infact, IIRC the luasocket MIME routines are locked away in the SMTP section, and can't readily be used by HTTP clients. 2016-01-27 13:49:42 TheCycoONE I think the WAP Push spec requires multipart/mixed 2016-01-27 13:49:43 zash daurnimator: eh, just require"mime"? 2016-01-27 13:49:53 zash WAP? Hahahahaha 2016-01-27 13:50:01 daurnimator TheCycoONE: WAP is something else entirely 2016-01-27 13:50:10 furq oh nvm that's in XHR 2016-01-27 13:51:07 furq even so i doubt it's worth your time supporting that on the server side 2016-01-27 13:51:15 zash Yet I still have no idea how to respond to a form POST with 2xx without the browser re-posting if you refresh the page... 2016-01-27 13:51:20 furq even if you've already written a multipart encoder 2016-01-27 13:51:46 daurnimator zash: the multipart stuff is locked up behind smtp.message() 2016-01-27 13:51:51 daurnimator zash: use a 303. 2016-01-27 13:52:30 TheCycoONE zash, PRG pattern (https://en.wikipedia.org/wiki/Post/Redirect/Get) - what daurnimator said 2016-01-27 13:52:34 zash daurnimator: but that's not very RESTful 2016-01-27 13:52:38 TheCycoONE ah 2016-01-27 13:52:56 rjek What is this, hipster central? Get the fuck out with your REST 2016-01-27 13:53:00 zash also already did that http://hg.prosody.im/issue-tracker/rev/cd49d6c23da3 2016-01-27 13:53:10 TheCycoONE use PUT in rest for things that you don't want duplicated 2016-01-27 13:53:24 TheCycoONE and then handle it with the id 2016-01-27 13:53:54 zash rjek: I'm using RESTful ironically! 2016-01-27 13:54:04 TheCycoONE but why is a web browser interacting with REST directly? 2016-01-27 13:54:49 furq anyway 2016-01-27 13:55:23 furq daurnimator: beyond request(url, { file = handle }) and request(url, { file = { handle = handle, type = "image/png" }) i don't care what you do 2016-01-27 13:55:32 furq +} 2016-01-27 13:56:11 furq i'm pretty sure that will cover 99% of use cases anyway 2016-01-27 13:57:09 furq and by 99% i of course mean by number of requests made, not number of potential bullshit rfc-says-SHOULD uses 2016-01-27 13:57:13 daurnimator furq: how about: myrequest:set_multipart_body(function() local h = new_headers(); h:append("content-type", "image/png"); h:append("content-disposition", "form-data; name=file"); coroutine.yield(h, handle) end) 2016-01-27 13:57:41 furq that seems like the kind of thing you could magic up for me when i pass a file handle 2016-01-27 13:58:46 daurnimator furq: I can't inger the file name; and I don't want to infer the mime type :p 2016-01-27 13:58:49 daurnimator s/inger/infer/ 2016-01-27 13:59:04 furq the mime type should default to application/octet-stream 2016-01-27 13:59:13 furq and the name should be whatever the key is 2016-01-27 14:00:04 daurnimator furq: the alternative is some sort of multipart/form-data construction class..... local mb = new_multipart_body(); mb:add_form_data("myname", "image/png", handle); request:set_body(mb); 2016-01-27 14:00:41 furq is that how you add form data anyway 2016-01-27 14:00:52 furq minus the new_multipart_body ofc 2016-01-27 14:01:52 furq also if you do it that way then new_multipart_body should take a table of initial form-data values 2016-01-27 14:02:06 furq as well as add_form_data 2016-01-27 14:02:22 daurnimator ehhhhhh 2016-01-27 14:03:05 furq new_multipart_body({ name = { handle, "image/png" } }) is much less hassle 2016-01-27 14:03:34 daurnimator furq: problem is I want to imply order (cause the order large things are sent tends to matter) 2016-01-27 14:03:58 furq if you want a defined order then use add_form_data 2016-01-27 14:04:03 furq and by you i mean the user 2016-01-27 14:05:00 furq http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file 2016-01-27 14:05:11 furq it would be nice if it was this simple, although i'm aware this is a high-level api wrapper 2016-01-27 14:05:26 furq maybe you could separate that stuff into a helper libary 2016-01-27 14:05:47 daurnimator furq: helper libraries are hard when you want them to be methods on existing objects.... 2016-01-27 14:09:07 furq also you should definitely infer the content-type as "text/plain" for string form-data values 2016-01-27 14:09:22 furq that is the official(tm) default 2016-01-27 14:10:17 furq actually 2016-01-27 14:10:44 furq As with all multipart MIME types, each part has an optional "Content-Type", which defaults to text/plain. If the contents of a file are returned via filling out a form, then the file input is identified as the appropriate media type, if known, or "application/octet-stream". 2016-01-27 14:11:00 furq there are defaults defined for files and strings 2016-01-27 14:11:49 furq also nice grammar "l. masinter"

[daurnimator]

also requested by @ner0x652 for https://www.virustotal.com/de/documentation/public-api/

[daurnimator]

Worth looking at the python requests api for this: http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file

Or their advanced streaming api: https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-multipart-data-encoder

[daurnimator]

Browsers now have FormData API that could be copied as well.