How can refs help in figuring out which response belongs to which request?

[massivefermion]

I'm trying to port Mint to Gleam, but there are things I don't fully understand. Apparently you use refs to figure out which received tcp packet belongs to which request, but I don't see how that can work given that the tcp packets you receive don't contain the ref. How can you associate those packets with a request if they don't have any ref in them? I've tried reading Mint's code, but it seems like in the stream function, it's just putting whatever the current request's ref is in the response. I'm really confused!

P.S. For now just focusing on http1.1

[the-mikedavis]

HTTP/1 doesn't support multiplexing so the reference will always belong to the request sent on that connection. It's a little more complicated if you're using HTTP/1.1 pipelining: the responses must be sent by the server in the order that the requests were received so the Mint.HTTP1 struct has a queue in the :requests field. See: https://github.com/elixir-mint/mint/blob/fe80d7ec248763f7d912acd3c52f39655f4d9acd/lib/mint/http1.ex#L872-L892

In HTTP/2 the references correspond to the stream IDs of the packets.

[whatyouhide]

@massivefermion the raw TCP packets themselves do not contain any ref, that's correct. However, once you parse them, they contain data that allows us to identify the request they belong to.

• For HTTP/1.1 (with pipelining), then requests are in a queue so it's easy to reference them back: once a response is parsed, the data after it belongs to the next request in the queue.
• For HTTP/2, frames contain a stream ID (both requests as well as responses), so we can use the stream ID that we parse out of inbound TCP packets to map it back to the corresponding request (and thus fetch its ref).

Hope this clarifies it!

Bonus question: by "porting" this to Gleam, to you mean rewriting the code? Is it something that's needed to make this work in Gleam? cc @lpil, I had a feeling Gleam made it pretty easy to use Elixir packages, right?

[lpil]

Hello! Yes, Gleam can use Elixir packages, but Elixir is a huge dependency for a HTTP client, and because it and the stdlib cannot be distributed to Hex depending on it adds additional complexity to the build process. Projects failing to build or run due issues with Elixir is one of the two most common categories of build problems that get reported. The other is that apt install erlang doesn't include any OTP dev applications so EUnit fails.

[whatyouhide]

@lpil ah okay that makes sense. That's a bummer because Mint is a BIG package with really a bunch of complexity (especially in the HTTP/2 part), so hopefully the port will work well!

[massivefermion]

@lpil ah okay that makes sense. That's a bummer because Mint is a BIG package with really a bunch of complexity (especially in the HTTP/2 part), so hopefully the port will work well!

definitely true, I was really hesitant to get into it. Hopefully given enough time, I can make it work.

[lpil]

Yeah it's a shame! Me and Erlang folks often wish that Mint was written in Erlang like Telemetry is, but that's life 🤷

[wojtekmach]

Ideally Mint would have been OTPs gen_http, gen_http1, and gen_http2 modules (and similarly to gen_tcp you could use these to write servers) but that's waaaay easier said than done.

[wojtekmach]

Elixir is a huge dependency for a HTTP client, and because it and the stdlib cannot be distributed to Hex depending on it adds additional complexity to the build process

Right, elixir, mix, and friends are reserved names on Hex.pm.

I don't know if this solution has been explorer but, similarly to projects built with rustler_precompiled, I think you could do the following:

1. Create gleam_mix project.
2. If you're unable to run arbitrary code when building gleam projects, probably best to build this with a Makefile

3. It's easier if the package has hardcoded Elixir version but it's not that much harder to support arbitrary ones. Assuming it's hardcoded, add a checksums.txt file to your package with checksums per OTP version. You can grab these from:

   • https://github.com/elixir-lang/elixir/releases/download/v1.15.7/elixir-otp-24.zip.sha256sum
   • https://github.com/elixir-lang/elixir/releases/download/v1.15.7/elixir-otp-25.zip.sha256sum
   • https://github.com/elixir-lang/elixir/releases/download/v1.15.7/elixir-otp-26.zip.sha256sum
4. when building package download Elixir for the current OTP major version from https://github.com/elixir-lang/elixir/releases/download/v1.15.7/elixir-otp-26.zip and similar URLs
5. Compare downloaded zip against checksums.txt
6. Unpack zip and put it into some _build folder equivalent. I believe it's fine if one package contains multiple .app files but haven't tested it.
7. Create gleam_mint that depends on gleam_mix. In order to build Mix projects all we need is bin/elixir and bin/mix in the PATH so the build tool hopefully can handle that.

Apologies if I'm missing something very obvious but I think it can actually work and honestly doesn't sound too bad, I believe downstream packages shouldn't need to build Elixir from source anyway.

[wojtekmach]

cc @tsloughter

[whatyouhide]

Yeah it's a shame! Me and Erlang folks often wish that Mint was written in Erlang like Telemetry is, but that's life 🤷

Absolutely @lpil. We actually started writing it in Erlang, but it turned out quite cumbersome, especially without structs. Back then the intention was to potentially include Mint in Elixir's stdlib, so we ended up giving up on Erlang and just going with Elixir. So yeah, series of unfortunate events 😢

[tsloughter]

@whatyouhide curious how structs help so much? I get when a core library benefits greatly from macro usage (like postgrex does for performance) but hadn't considered structs.

[tsloughter]

I'm also considering proposing stipends for http work in OTP. Maybe that could be "gen_http".

I maintain a http2 server/client (chatterbox) and would prefer not to :), but yea, requiring the entire Elixir as a dependency installed by the user just isn't an option.

[whatyouhide]

I'm also considering proposing stipends for http work in OTP

I’m (obviously) into this, feel free to reach out if this ever becomes something.

I get when a core library benefits greatly from macro usage

We do this too 😄

curious how structs help so much?

Just easier to delegate to the right modules between HTTP/1 and HTTP/2, mostly. Plus I mean, the compilation-time checking of fields is also pretty nice 😉

[lpil]

@wojtekmach Having it automatically install before is something I've vaguely wanted but I've not put at much thought into it as you have here, so thank you for that! I'll make an issue to keep note of it.