UNIX domain socket support

[jgillich]

I would to to access an UNIX socket with hyper. TcpStream and UnixStream are very similar, but I assume it would still require some heavy changes (or, looking at net::HttpStream, maybe not?). So before I waste any time on this, do you guys think it would make sense to have this in hyper?

[seanmonstar]

The desired goal is to utilize the HTTP protocol over a Unix stream instead of a TCP stream? It should mostly just need an implementation of net::NetworkStream/NetworkConnector. Not sure if the use of Url will cause any issues...

[reem]

It should be pretty simple, as @seanmonstar said all you will need to do is implement the appropriate traits and use listen_network.

[seanmonstar]

As for whether it makes sense in hyper, I don't know. How about other languages/httplibs? Do they include a Unix stream option?

Unless they do, my gut feeling is this could be a separate crate, so as to not add more maintenance required for hyper.

[jgillich]

It would need a Path in addition to Url, but there's still a totally normal Url where the host part gets ignored.

FYI, it would be useful for accessing the Docker remote API, where the Unix socket is the default and also the only standard way to limit access (they don't do authorization over HTTP as far as I know). I am not sure how many other programs do HTTP over Unix sockets. probably not a lot.

About libraries that support it, Node's built-in http library does and some libraries that use it also mention it explicitly (for example request).

I don't mind putting it in a separate crate, it surely is a uncommon feature. It will be a while before I start implementing this anyway, I think it can be decided when it is a bit more clear how the required changes would look like.

[jimmycuadra]

I am also interested in this for use with the fleet API. At least some Ruby HTTP clients support requests to UNIX sockets, e.g. Excon.

[softprops]

This may be of interest to you folks. I just published a new crate called hyperlocal which provides hyper interfaces for unix domain sockets. I realized that while working on a docker client which also supports unix domain sockets that I could support both tcp and domain socket interfaces using hyper's client interface. I'll be rolling hyperlocal into that soon to replace the hand crafted http unix domain socket interface I'm currently using.

[jimmycuadra]

That's awesome, softprops! Thank you.

[jgillich]

Very cool :) I guess this can be closed then.

[softprops]

I really want to emphasize how well designed hyper's internals were made. It made it very straight forward to extend it in this way. +100 @seanmonstar

[valpackett]

Would be nice if hyper itself could listen on unix sockets, without an additional crate.

It also should support listening on user-provided (already opened) sockets (for both tcp and unix), e.g. acquired via socket activation or file descriptor passing.

Actually I think socket activation should be handled in hyper itself as well… even though there's a brilliant pun opportunity for naming a crate that does it ;-)

[prepor]

It's an essential feature for http client library, most popular http client libraries support it. We also have already have support of unix sockets in tokio (https://github.com/tokio-rs/tokio-uds). Is there a chance that such feature will be accepted in case of PR?

[seanmonstar]

I wouldn't expect hyper to explicitly include a UNIX socket implementation. It's already possible to just use any external implementation by using a custom Connect in the Client. hyper's goal is provide HTTP, and be generic over whatever the transport is underneath.

Being built in to a client would make sense for a more batteries-included kind of library, and for this exact feature, see https://github.com/seanmonstar/reqwest/issues/39

[softprops]

FYI, I'm planning on adapting to hypers interfaces for uds in https://github.com/softprops/hyperlocal as soon as there's a new release for hyper

[valpackett]

hyper's goal is provide HTTP, and be generic over whatever the transport is underneath

Well, it has a bind method that starts a TCP listener…

Looking at the master documentation, bind_connection seems like the right API though :)

[prepor]

@seanmonstar ok, hyperlocal of @softprops looks good, waiting for a release ;)

[ghost]

@seanmonstar, can you please reopen this? I wanted @softprops's hyperlocal server to support run_until so I implemented it. The file is almost entirely copy+pasted from a subset of this one in hyper but with UnixListener instead of TcpListener. I think it would make a lot more sense for hyper::server::Server to just have a type argument for listener instead of TcpListener.

To avoid breaking changes perhaps we could have a Server2<S, B, L> with listener: L and Server<S, B> = Server2<S, B, TcpListener> or something to that effect.

[seanmonstar]

I do expect Server to have a generic type for the incoming stream in 0.12. On master, there is Serve which is already this, and will likely evolve and replace the current Server as the tokio reform stuff is completed.

I don't think this issue itself needs to be opened for that, as this was around building in UDS support by default, instead of allowing it to be used.

[softprops]

Hi there. I'm open to prs for if anyone needs a change https://github.com/softprops/hyperlocal/blob/master/README.md

[ghost]

I find it strange and surprising that Hyper doesn't support Unix domain sockets. It's a basic and very well-supported feature in any descent programming language (i.e. not Java). Node supports it. Python supports it. I use it a lot for running various services proxied behind Nginx. The crate hyperlocal depends on an old version of Hyper. If I were to change to TCP sockets I can't even set permissions. Please reopen.

[valpackett]

Yeah. And looks like there's still no easy way to provide a custom TcpListener/UnixListener (e.g. from listenfd).

[olix0r]

Now that Hyper uses a recent Tokio, it should definitely be possible to instrument Hyper around tokio-uds. Whether this belongs as a part of Hyper or just a recipe that can be shared, i have no opinion.

[sfackler]

@myfreeweb

unix_listener.for_each(|(socket, _)| {
    let connection = Http::new().serve_connection(socket, MyService);
    tokio::spawn(connection);
    Ok(())
});

[seanmonstar]

@portstrom Both hyper's client and server allow providing any custom connector or listener, so you can plug in UDS easily, as others have shown.

Does it really seem so common to use UDS that it should actually be built in to hyper directly? If it is, at this point, it wouldn't be that much work to provide a hyper::client::connect::UdsConnector and a listener for the server. If it seems common enough, @portstrom would you be interested in adding it, so others aren't "surprised" in the future?

[softprops]

I have a crate that supports tokio uds in the meantime https://github.com/softprops/hyperlocal

[seanmonstar]

Thinking more, it really doesn't seem that common. I'm fine with it being a separate thing.