How to upload files with multipart/form-data?

[xuchunyang]

For example, curl -F file=@1.txt -F file=@2.html localhost:3000 sends these to the server

$ nc -l localhost 3000
POST / HTTP/1.1
Host: localhost:3000
User-Agent: curl/7.64.1
Accept: */*
Content-Length: 346
Content-Type: multipart/form-data; boundary=------------------------a0a877124b35be18

--------------------------a0a877124b35be18
Content-Disposition: form-data; name="file"; filename="1.txt"
Content-Type: text/plain

2020-12-07

--------------------------a0a877124b35be18
Content-Disposition: form-data; name="file"; filename="2.html"
Content-Type: text/html

<h1>hello</h1>

--------------------------a0a877124b35be18--

the following works, however it is hard to write

(setq
 foo
 (mm-url-encode-multipart-form-data
  '(("file" . (("name" . "file")
               ("filename" . "1.txt")
               ("content-type" . "text/plain")
               ("filedata" . "2020-12-07")))
    ("file" . (("name" . "file")
               ("filename" . "2.html")
               ("content-type" . "text/html")
               ("filedata" . "<h1>hello</h1>"))))
  "------------------------a0a877124b35be18"))

* test                                                                 :verb:
template http://localhost:3000

** upload files
POST /
Content-Type: multipart/form-data; boundary=------------------------a0a877124b35be18

{{foo}}

[federicotdn]

Hey @xuchunyang, thanks for bringing this up. Unfortunately this is not implemented yet, as I've never had to use multipart yet.

I think this is an interesting implementation challenge because one of the ideas of Verb is that the request specifications one writes should be as close as possible to what is actually sent afterwards. For multipart, this clearly would not be practical.

If you have any ideas/code, feel free to share. Currently I don't have time to look into it, but I might in a month or two.

[ercdude]

tnks @xuchunyang for the workaround. I'm dealing with this problem right now and I'm trying to make a function to help with it. Once I find out how to use it with binary data, I try to add something to verb and share with you.

In case anyone has done it already, please share how did you read a binary file and inserted its data to the requisition. My first thought is to read it raw and insert it, but I'm afraid of big files lol

[federicotdn]

Would something like this work for you? I'm trying to find a balance between something that's not too manual, but doesn't try to enforce a specific structure:

Version 1:

* Example request                        :verb:

post www.example.com
Accept: */*
Content-Type: multipart/form-data; boundary={{(verb-boundary)}}

{{(verb-boundary)}}
Content-Disposition: form-data; name="file"; filename="1.txt"
Content-Type: text/plain

{{(verb-read-file "/path/to/1.txt")}}
{{(verb-boundary)}}
Content-Disposition: form-data; name="file"; filename="2.html"
Content-Type: text/html

{{(verb-read-file "/path/to/2.html")}}
{{(verb-boundary t)}}

Summary:

• The verb-boundary function automatically returns the value of the boundary. It can optionally be set in the heading Org poperties (Verb-Boundary: ------abcdef), otherwise it will be automatically generated.
• The user has to insert the boundaries by themselves. At the end of the request, t must be passed to the verb-boundary function to signal that it is the final boundary.
• The contents of the files themselves are read using verb-read-file (note that this function already exists in Verb).
• Properties such as name or filename, or other headers (e.g. Content-Type) must be provided by the user as well.

[federicotdn]

Version 2:

* Example request                        :verb:

post www.example.com
Accept: */*
Content-Type: multipart/form-data; boundary={{(verb-boundary)}}

{{(verb-part "file" "1.txt")}}
Content-Type: text/plain

{{(verb-read-file "/path/to/1.txt")}}
{{(verb-part "file" "2.html")}}
Content-Type: text/html

{{(verb-read-file "/path/to/2.html")}}
{{(verb-part)}}

Summary:

• The verb-boundary function automatically returns the value of the boundary. It can optionally be set in the heading Org poperties (Verb-Boundary: ------abcdef), otherwise it will be automatically generated.
• Boundaries and Content-Disposition headers can be inserted using the verb-part function. The function has two parameters: NAME and FILENAME (both optional).
• Like in Version 1, additional headers such as Content-Type must be added manually by the user.
• Contents of the file are also inserted using verb-read-file.
• The final boundary is inserted by calling verb-part with no arguments.

[ercdude]

@federicotdn nice! this is basically what I'm doing here, but I'm defining these variables manually:

  (setq-local post-body
              (mm-url-encode-multipart-form-data
               '(("name" . "asdsad...")
                 ("key" . "oasdasd=="))
               http-boundary-token))

*** post
post /
Content-Type: multipart/form-data; boundary={{http-boundary-token}}

{{post-body}}

Using that function seems to help a lot as it solves the boundary= and the Content-Disposition internally. I liked that :)

[federicotdn]

Implemented and documented in https://github.com/federicotdn/verb#multipart-data ! Feel free to open a separate issue if something could be improved or fixed.