Is there a way to use a unix socket?

[bbigras]

I would like to do the equivalent of:

curl --unix-socket /var/run/docker.sock http://localhost/containers/json

There's the unix_socket crate but I'm not sure if I can use it with reqwest.

[seanmonstar]

There isn't a way yet, but it definitely makes sense to provide when cfg(unix)!

What makes sense for the best way to provide this? Just adding to the connector to look for the unix:// scheme? There doesn't seem to be a standard way to handle this, as I look up various HTTP libraries.

[stephanbuys]

The unix:// scheme seems to be well established, I'd recommend doing what you propose.

[bbigras]

The url crate seems to support the unix:// scheme (see example).

let url = Url::parse("unix:/run/foo.socket").unwrap();

Would it be possible to include the path with something like this?

let url = Url::parse("unix:/var/run/docker.sock/containers/json").unwrap();

I'm also wondering if localhost is passed to the unix socket. If so, I don't know how it would be possible to specify it (or another vhost) using the unix:// scheme.

According to the Docker Remote API page, older curl version used http://containers/json as the URL so I don't know if the localhost part is used.

[seanmonstar]

So looking around, I see two possible routes.

• As curl does it, which is to not use the URL to indicate that a unix socket should be used instead of TCP. This could either be a single constructor on the client, like Client::unix_socket("/var/docker.sock"), or it could be considered a "proxy", and thus as suggested in #30, be something like Proxy::unix("/var/docket.sock").
• Using the URL, something like requests-unixsocket, with an example of http+unix://%2Ftmp%2Fprofilesvc.sock/path/to/page. This is a whole lot uglier, but allows all the semantics to exist in the Url, such as whether to use TLS, and is possibly easier to allow dynamic selection of which unix socket.

[lucab]

@seanmonstar in case you were looking for feedback, I definitely prefer the first option because it allows to leave the scheme and host-fqdn part untouched. This is important if at the other end of the unix-socket there is a virtualhost-aware server, or something doing TLS-ALPN (or similar upper protocol detection/negotiation).

[mathstuf]

I've wanted "use a socket for HTTP requests" for a while because setting up a proxy on a port doesn't lock it down to just the current user. Not that that project is ever going to use Rust at its network level anytime soon, but it's not an unheard of use case.

[seanmonstar]

I think I prefer this being Proxy::unix(path). For implementation details, it'd need to be slightly different since it would actually connect a unix stream instead of just give a new URL.

[farodin91]

Any progress in this area?

[mathstuf]

Doing some research in how it is done elsewhere:

• Apache HTTP it looks like fcgi://socket=%2ftmp%2fphp-fpm.sock/local/htdocs/ is proposed. Though it seems that actual support may be unix:///var/run/php-fpm.sock|fcgi://127.0.0.1:9000/srv/www/$1
• NodeJS uses named options.
• Here is my libsoup thread about it. (The thread continues into February as well.)

[farodin91]

I saw a implementation of unix socket in boondock using hyper 0.9.0 https://github.com/faradayio/boondock/blob/master/src/unix.rs

[mathstuf]

This also might not be a non-Windows thing in the future as well: https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

[jjl]

I came across the hyperlocal crate today which might be of interest

[Roguelazer]

Is there any workaround now? I was looking at something with the xmlrpc crate (which uses reqwest by default) but unix sockets are a must-have for my environment and it looked like all the Transport stuff in reqwest is currently hidden away in the async_impl module; there didn't seem to be any hooks in ClientBuilder where I could shove a custom Transport.

[doctorCC]

Has there been any progress on this? I am also looking for a way to use the Docker authorization via the Docker socket, but without going through the loopback address.

[snawaz]

@seanmonstar .. I've designed a solution to support Unix Socket... and we already use it in our codebase. I can make it more generic and reqwesty .. and then submit a PR, in case if you're not already working on it. Please let me know your thoughts.

[scottlamb]

@seanmonstar To be precise, I think you mean something like Proxy::all("unix:/var/run/docker.sock") right? as Proxy::http vs Proxy::https vs Proxy::all vs Proxy::custom appears to be about what should be proxied, not how it's proxied. Looks like the argument determines the latter, within ProxyScheme::parse.

[lucasyvas]

It would be nice if this could work on Windows too via named pipes.

[asf-stripe]

I found this discussion after trying to use reqwest with our unix domain socket-based authentication proxy, and was reminded of this: One of the things that I find that go's http.Transport got right is the way you establish an HTTP-able connection. The Transport has Dial (and, because it's go, DialContext) field that holds a function, which establishes the unencrypted TCP connection.

I kind of wish that reqwest allowed this kind of customization: I'd like to be able to tell the system in what way to establish a network connection. Then, reqwest could be entirely independent of how the proxy wants to be reached: named pipe, unix domain socket, CONNECT proxy over open SMTP relay, etc.

[zicklag]

I would also use this if it was available. I'm making a REST API, but I need to be able to access it by mounting a Unix socket into a container. Reqwest seems pretty de-facto for making HTTP requests so it would be really nice to be able to use it for Unix sockets too.

[mathstuf]

I'll also note that Docker's socket talks HTTP.

[vv9k]

What's the current status of this? As @snawaz mentioned it's already partially ready. Would love some updates as I'd love to talk to docker over a unix socket. Cheers :)

[danieleades]

This is really easy to do with Hyper. Any chance of exposing the necessary methods? Or better yet, creating a Request Client from a Hyper Client?

[mathstuf]

I'm not seeing anything obvious in the docs. Do you have a link to the methods involved (or example code)?

[danieleades]

I'm not seeing anything obvious in the docs. Do you have a link to the methods involved (or example code)?

if you were talking to me, you could take a look at my pull request on hyperlocal for some code examples

[kevincox]

I would recommend the unix: scheme as well because it means that all reqwest users would get it automatically (or with a feature enabled). It would also mean that all tools that use reqwest would have a consistent syntax for specifying UNIX sockets.

[Absolucy]

with the hyper client, it's easy to send http requests over anything that implements AsyncRead and AsyncWrite

let (mut request_sender, connection) = hyper::client::conn::Builder::new()
    .handshake::<MyCustomTcpStream, Body>(stream)
    .await?;

let request = Request::builder()
    .header("Host", "example.com")
    .method("GET")
    .body(Body::from("hello world!"))?;

let response = request_sender.send_request(request).await?;

[ghost]

Would be nice if it was possible to use a unix domain socket as a socks5h proxy (tor provides a socks uds option for example)

[jvjfk]

Hi everyone. Any update on this issue? Since it has been opened for 4 years I wonder if there is anything to do to help :)

[ahl]

This would be a great addition!

[Kijewski]

Apache's ProxyPass uses the syntax unix:/path/to/socket|http://example.com/path?query, i.e. socket path + pipe + URI, which I think is very readable, and has the advantage that you can use it with HTTPS, basic authorization, etc.

[insanitybit]

It seems like there's an open PR that would solve this. Any change to have this merged in?

[bchalios]

It would be really nice to see this PR merged.

[insanitybit]

For anyone running into this, I've found that the hyper-socket crate requires very few changes to work with the latest hyper/ tokio.

https://docs.rs/hyper-socket/latest/hyper_socket/

Doesn't solve the issue for reqwest, but still, it's something.

[pythops]

Would be really great to have :pray:

[Mart-Bogdan]

Would be nice to have.

I agree with @asf-stripe that instead of file name, we can provide callback that creates socket, it can be even linux File descriptor. And on top of it create helper method that accepts file name as String.

Theoretically it can even be some kind of rust stream instead of raw FD, but I don't know inner workings of library to judge.

If out can be some stream, then even TLS part can be build on top of this.

[Ambyjkl]

Here is the format used by nginx, which i think is better than anything unix://: http://unix:{sockfile_path}:/{pathname}?{query}. The protocol is http/https, just instead of an IP address or DNS hostname, it's a unix socket file path

[Mart-Bogdan]

@Ambyjkl you still need to provide domain name somehow.

Domain name is used not only to resolve IP address, but it is also transferred as Host: header.

It can be used by server, if two websites are living on same socket.

And some servers won't return data for incorrect host.

[afidegnum]

So, can i use reqwest with Unix sockets? what's the basic syntax?

[7ritn]

I was struggeling with getting this to work as well. I wanted to perform curl --unix-socket /run/user/1000/podman/podman.sock http://d/v4.0.0/libpod/containers/lldap/logs?stdout=true in rust. I finally got it to work with hyper and axum.

use axum::{
    body::Body,
    http::{Method, Request, StatusCode}
};
use http_body_util::BodyExt;
use hyper_util::rt::TokioIo;

use std::path::PathBuf;
use tokio::net::UnixStream;

pub async fn client(container_name: String) {

    let path = PathBuf::from("/run/user/1000/podman/podman.sock");

    let stream = TokioIo::new(UnixStream::connect(path).await.unwrap());
    let (mut sender, conn) = hyper::client::conn::http1::handshake(stream).await.unwrap();
    tokio::task::spawn(async move {
        if let Err(err) = conn.await {
            println!("Connection failed: {:?}", err);
        }
    });

    let request = Request::builder()
        .method(Method::GET)
        .uri("/v4.0.0/libpod/containers/lldap/logs?stdout=true")
        .header("Host", "d")
        .body(Body::empty())
        .unwrap();

    let response = sender.send_request(request).await.unwrap();

    assert_eq!(response.status(), StatusCode::OK);

    let body = response.collect().await.unwrap().to_bytes();
    let body = String::from_utf8(body.to_vec()).unwrap();
    println!("{}", body);
}

#[tokio::main]
async fn main() {
    client("lldap".to_string()).await;
}

[cablehead]

This is a small example using just hyper (no need for axum), to make requests to a local unix domain socket: https://github.com/cablehead/stacks/blob/main/src-tauri/src/cli.rs#L30

Dropping back to raw hyper though, you lose, http1.1 / http2 negotiation, ssl, and connection pooling.

Although 🤔 .. those things are generally less necessary for the use cases where you'd find yourself wanting a local socket..

[danieleades]

there have been several solutions posted for solving this in different ways using hyper going back as far as 2018. This thread is specifically about using reqwest with a unix socket

[sergi0g]

Is there any update on this?

[kczimm]

+1

[kanpov]

How much work would be necessary to make reqwest fully generic over any type implementing hyper::rt::Read and hyper::rt::Write? This would allow not only unix sockets via tokio's net module, but also vsock and so on.

[mikibella]

Any updates on this topic?

[frankdavid]

We'd also be interested in using reqwest with a custom Connector. We'd like to customize how the TCP/TLS connection is built up but use reqwest for HTTP communication.

[sergi0g]

This is the oldest issue in this repository (8 years old) and it's a shame it's still open. So many people are showing their support and we're still waiting for an answer. The last comment from the maintainer was 7 years ago! A "yes we're working on it" or "no it won't be implemented" is enough.

[Ambyjkl]

What I'd like to see: being able to run reqwest over an arbitrary AsyncRead + AsyncWrite, where one can pass in one of our own and just manually handle connecting to stuff like unix socket. I believe hyper lets you do this, so just exposing it to reqwest would be great