(socks) proxy support

[Stebalien]

In some environments, users need to be able to proxy all traffic through, e.g., a corporate socks proxy. This sucks but is sometimes inevitable.

Solution:

• Tunnel outbound connections through the socks proxy.
• If the proxy supports bind use it.
• Otherwise, connect to and advertise a relay address.

Problems:

• Proxy aware transports are going to be a royal pain.

[Stebalien]

So, @rklaehn pointed out a simple solution for dialing: use the websocket transport. It should work just fine over any HTTP(s) proxy. All we need to do is expose this somehow.

Unfortunately, listening could be tricky.

[rklaehn]

So it should work, but in my experiments I could not actually get it to work. Just to clarify: websocket is a complete transport that will work just as well as the native tcp or udp transports? So in theory you could have a swarm just communicating over ws? Or is it in some way limited?

[Stebalien]

Yes, it's a full transport. I assumed it wouldn't work because we don't explicitly tell it to use the proxy from the environment. However, it looks like it does this by default.

So, this is a bit odd. It should just work. I'm wondering if it's a port issue. Does your proxy not support forwarding to random ports?

[cretz]

I too need a transport proxy interface. In many cases in Go, proxies are implemented by accepting a net.Dialer and/or a net.Listener. There needs to be some way to downgrade a transport to these items or the logic that does things with the connections needs to be decoupled from the logic wrapping in manet dialier/listener and transports. My use case is for web sockets over Tor (which happens to be a socks proxy for dialing, but there is also a part I want to proxy for listening). However, there is enough unexposed logic in the websocket that I have to take the entire websocket transport dial/listen or none of it. I would prefer just the parts that operate on the net connections, or even better, let it take an externally provided tcp dialer/listener.

EDIT: as mentioned in https://github.com/libp2p/go-ws-transport/issues/33 I instead am just proxying raw net.Dialer/net.Listener stuff instead of proxying an entire transport. With all the things that transports add, I am thinking that proxying at the transport level is less practical than at the raw level and then wrapping.

[ConorTighe]

Did anyone find a solution that worked for this problem?

{"error":"failed to bootstrap. context deadline exceeded","event":"bootstrapError","peerID":"QmXL3Pe6USjNf5Vks2XBi7ryngCe8NguazLyCjeeeCVVe3","system":"core","time":"2018-07-17T09:32:28.447604435Z"}

[Stebalien]

So, I've tested this and dialing does actually work. You just need to correctly configure your environment to use a proxy: https://wiki.archlinux.org/index.php/Proxy_settings#Environment_variables

For example, if you're using SSH, you can run http_proxy=socks5://localhost:PROXY_PORT ipfs daemon and all outbound websocket connections should just use this proxy.

Note: this still doesn't help with inbound connections.

[ConorTighe]

I have my environment variables set in etc/environment and exported http_proxy in ~/.bashrc, Ive tried passing http_proxy before the daemon and before the computes daemon. Im only connecting 2 Ubuntu machines through a bridged network and have them communicate in a swarm. Im still getting the following task status:

{"meta":{"computes":{"queue":{"assigned":[{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T08:37:09+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-16T11:01:20+01:00"}],"available":[{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-16T11:43:58+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T11:08:53+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T11:17:32+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T08:25:58+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-18T10:37:48+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-18T10:19:05+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-18T10:24:42+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T10:57:44+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-16T11:01:19+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T08:37:09+01:00"}],"completed":[{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-16T11:01:22+01:00"},{"hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T08:37:12+01:00"}]},"runner":{"errors":[{"error":"UpdateResultsBytes failed: Unable to update interface: DAG.PutInterface failed: Put failed: unexpected status code returned. Expected 200, received 500","hostname":"ctighe-VirtualBox","timestamp":"2018-07-17T08:37:12+01:00"},{"error":"UpdateResultsBytes failed: Unable to update interface: DAG.PutInterface failed: Put failed: unexpected status code returned. Expected 200, received 500","hostname":"ctighe-VirtualBox","timestamp":"2018-07-16T11:01:22+01:00"}]},"seed":"word-status","tasks":{"related":["zdpuAz6fL6221SDcEdVPbr8M3KYKsL1bHjQcxX8WB8shZ6yBa","zdpuAv1doragWYqB2EKC3cL33zEiqFi3J9M1oQJBtAW7vJJHK"]}}}}

[Stebalien]

Ah. Sorry, I forgot. While we can dial websocket addresses, go-ipfs doesn't listen on them by default. That's probably the problem here.

Note: we haven't enabled listening by default as our primary usecase for the websocket transport is allowing browsers to dial go-ipfs nodes. Unfortunately, browsers often need to dial websocket over https and IPFS nodes can't get valid certificates.

I've opened an issue for enabling it by default: https://github.com/ipfs/go-ipfs/issues/5251.

In the mean-time, try running ipfs swarm connect /dns4/ams-1.bootstrap.libp2p.io/tcp/80/ws/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd. That'll connect to to one of the bootstrappers we usually reserve for browser nodes.

[ConorTighe]

ipfs swarm connect /dns4/ams-1.bootstrap.libp2p.io/tcp/80/ws/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd

 Error: connect QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd failure: dial attempt failed: <peer.ID XL3Pe6> --> <peer.ID SoLer2> dial attempt failed: context deadline exceeded

[Stebalien]

Damn. It looks like the bootstrappers only listen on wss (websocket over https) and go can't dial that. Unfortunately, the PR to implement that is stale.

[ConorTighe]

Any idea what the scope might be to update this?

[nickname76]

Are there any updates?

[ynkumar143]

Team, are there any updates on this?

[clarkmcc]

Any updates on this?

[MarcoPolo]

Dialing wss is now supported (https://github.com/libp2p/go-ws-transport/pull/115). So this should work if you use the websocket transport

[pyhedgehog]

Are there bootstrap websocket nodes?

[aschmahmann]

Are there bootstrap websocket nodes?

How are you hoping to use this? If you're trying to leverage the IPFS Public DHT while only having websockets available as a transport you'll be out of luck since most of the DHT server nodes aren't listening on websockets and so your queries won't really work even if you can talk to the bootstrappers.