search

appcues / mojito

An easy-to-use Elixir HTTP client, built on the low-level Mint library.
https://hexdocs.pm/mojito/Mojito.html
MIT License
349 stars 34 forks source link

Feature request: Streaming request body? #79

Closed hubertlepicki closed 2 years ago

hubertlepicki commented 3 years ago

I have been dealing with weirdly large HTTP1 multipart requests in one of the projects I am working on, and this lead to hacking on Finch lib , where I have open an unmerged as of now PR https://github.com/keathley/finch/pull/107/files#diff-48431cc1d91063480b5006d7585c96ea39433e319aca2b5e3a6c597fdbd7e10fR145

I actually use that code in limited capacity so far on production and it seems to work well so far.

The objective is to be able to send large, multipart requests with files to an API that we use, which expects them, without loading whole files into BEAM memory. Think about these files being many megabytes in size, and multiple users uploading files at the same time. The server would, and did, use quite a bit of memory if it has to load them up to binary to stream first.

I had a look at Mojito and while it seems to be sending body in chunks, even on HTTP1 protocol (that I don't think needs to do, by the way - but maybe I'm wrong), it doesn't appear to work in case where body is a Stream itself.

My use case may be pretty specific, but I implemented the API where I can pass either String as a body parameter or a tuple of {:stream, stream}, not sure if the same API would be desired here.

While my mind is fresh and on the subject, I was wondering if you would like to have this also implemented in Mint? I could possibly work on a PR based on the Finch code I wrote earlier.

gamache commented 3 years ago

This certainly makes sense as a feature, and I like the idea of using {:stream, stream} as the input format. A PR would be very welcome!

hubertlepicki commented 3 years ago

OK. Quick, semi-related question: do you remember why this is here by any chance? https://github.com/appcues/mojito/blob/master/lib/mojito/conn.ex#L94

Is this some arbitrary choice made to send out data in chunks or is it some HTTP1 protocol requirement / implementation detail that makes a difference?

gamache commented 3 years ago

It's an arbitrary choice where HTTP is concerned. I did it that way so the HTTP/1 and HTTP/2 code looks similar.

ntenczar commented 2 years ago

Hi! This project is now deprecated:

We recommend that you use Finch which is also built on Mint. The creator of Finch has an excellent writeup here describing the problems with Mojito, and as a result we use Finch internally at Appcues now.