search

libp2p / go-libp2p-relay-daemon

[DEPRECATED] A standalone libp2p circuit relay daemon that made 2022 migration from V1 to V2 easier.
MIT License
26 stars 26 forks source link

Issues connecting from IPFS nodes over WSS #18

Closed TheDiscordian closed 1 week ago

TheDiscordian commented 2 years ago

Commit: a32147234644cfef5b42a9f5ccaf99b6e6021fd4

Issue:

Both js-ipfs and go-ipfs refuse to connect to this relay node over wss, works fine over ws.

What was tried:

Both go-ipfs and js-ipfs were tried, or simply: ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A:

Kubo 0.12 fails to connect:

$ ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
error: connect 12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A failure: no good addresses

To debug @lidel tried websocat:

$ websocat wss://ipfs.thedisco.zone:4430/ 
/multistream/1.0.0

But it indicates it should be working correctly. We also tried regular websockets, and those work fine. The reverse proxy server in use is Nginx.

Just in case, I tried to also update the cert, but this didn't change anything. It's worth noting that this is the exact same setup I was using with go-ipfs for relaying, and it was working fine. Config files provided below.

Configs:

config.json:

{
  "RelayV2": {
    "Enabled": false
  },
  "RelayV1": {
    "Enabled": true
  },
  "Network": {
    "ListenAddrs": [
        "/ip4/0.0.0.0/tcp/4011/ws",
        "/ip6/::/tcp/4011/ws"
    ],
    "AnnounceAddrs": [
    "/dns6/ipfs.thedisco.zone/tcp/4430/wss",
    "/dns4/ipfs.thedisco.zone/tcp/4430/wss"
    ]
  },
  "Daemon": {
    "PprofPort": -1
  }
}

/etc/nginx/sites-enabled/ipfs:

    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    upstream websocket {
        server 127.0.0.1:4011;
    }

map $remote_addr $proxy_forwarded_elem {
    # IPv4 addresses can be sent as-is
    ~^[0-9.]+$          "for=$remote_addr";

    # IPv6 addresses need to be bracketed and quoted
    ~^[0-9A-Fa-f:.]+$   "for=\"[$remote_addr]\"";

    # Unix domain socket names cannot be represented in RFC 7239 syntax
    default             "for=unknown";
}

map $http_forwarded $proxy_add_forwarded {
    # If the incoming Forwarded header is syntactically valid, append to it
    "~^(,[ \\t]*)*([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?(;([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?)*([ \\t]*,([ \\t]*([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?(;([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?)*)?)*$" "$http_forwarded, $proxy_forwarded_elem";

    # Otherwise, replace it
    default "$proxy_forwarded_elem";
}

    server {
        listen 4430 ssl;
        ssl_certificate /etc/letsencrypt/live/ipfs.thedisco.zone/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/ipfs.thedisco.zone/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
        location / {
            proxy_set_header Forwarded $proxy_add_forwarded;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_pass http://websocket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
        }
    }
TheDiscordian commented 2 years ago

Forgot to add, here's a minimal test for js-ipfs that returns the following error: Uncaught (in promise) AggregateError: No Promise in Promise.any was resolved

HTML:

<!DOCTYPE html>
<html lang="en">
<head><title>js-ipfs minimal relay test</title>

<script src="https://cdn.jsdelivr.net/npm/ipfs-core@0.15.2/dist/index.min.js"></script>

<script>
var ipfs;
async function main() {
    ipfs = await window.IpfsCore.create();
    await ipfs.swarm.connect("/dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A");
}
main()
</script>
</head>
</html>
marten-seemann commented 2 years ago

Why are you using a reverse proxy? The WebSocket transport now supports WSS.

TheDiscordian commented 2 years ago

@marten-seemann can you link me to docs for how to point it to my cert please? ❤️ I wasn't aware that I could ditch the reverse proxy now.

marten-seemann commented 2 years ago

Here's the option: https://github.com/libp2p/go-libp2p/blob/707100a521dace4a7a87fd17c04042ea483380b5/p2p/transport/websocket/websocket.go#L56-L62

In code, this would look something like:

libp2p.New(
    libp2p.Transport(ws.New, ws.WithTLSConfig(tlsConf)),
    ....
)
aschmahmann commented 2 years ago

@TheDiscordian you will not be able to connect to a WSS node from go-ipfs without doing a bunch of work (meaning custom builds with custom TLS verification) because go-libp2p could not dial WSS addresses authenticated by DNS addresses until we merged https://github.com/libp2p/go-libp2p/pull/1592 which was recent and therefore not yet released.

I ran vole.exe bitswap check bafkqaaa /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A (using https://github.com/aschmahmann/vole/pull/18) and got a connection just fine. There's a protocol not supported error but that's just because it doesn't speak Bitswap which is... correct for the relay 😄.

Without the PR above vole instead fails with the message below because of the WSS issue that was recently resolved: 

  * [/ip4/172.105.8.48/tcp/4430/wss] x509: cannot validate certificate for 172.105.8.48 because it doesn't contain any IP SANs
TheDiscordian commented 2 years ago

@marten-seemann Thanks! I'll work on ditching the reverse proxy today, and see if that narrows anything down (it'll be nice to axe Nginx)

@aschmahmann Ah thanks! This explains why go-ipfs failed, but not js-ipfs 🤔. Will try again with Marten's tip... (Edit: Or does the relay need bitswap for js-ipfs to be happy?)

TheDiscordian commented 2 years ago

I followed @marten-seemann's advice, you can see the code here: https://github.com/TheDiscordian/go-libp2p-relay-daemon/blob/master/cmd/libp2p-relay-daemon/main.go#L82

js-ipfs now returns a very unexpected error: Uncaught (in promise) Error: no protocol with name: "'dns'". Must have a valid family name: "{ip4, ip6, dns4, dns6}".

To confirm I'm trying the exact same thing: await ipfs.swarm.connect("/dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A");

/dns4/ returns the same error.

I also tested with websocat, and it seems to be happy (exact same result as previous).

Is this a bug with js-ipfs?

Edit: For completeness, here's my config.json:

{
  "RelayV2": {
    "Enabled": false
  },
  "RelayV1": {
    "Enabled": true
  },
  "TLS": {
    "KeyPairPaths": [["/etc/letsencrypt/live/ipfs.thedisco.zone/fullchain.pem", "/etc/letsencrypt/live/ipfs.thedisco.zone/privkey.pem"]]
  },
  "Network": {
    "ListenAddrs": [
        "/ip4/0.0.0.0/tcp/4011/ws",
        "/ip6/::/tcp/4011/ws",
        "/ip4/0.0.0.0/tcp/4430/wss",
        "/ip6/::/tcp/4430/wss"
    ],
    "AnnounceAddrs": [
    "/dns6/ipfs.thedisco.zone/tcp/4430/wss",
    "/dns4/ipfs.thedisco.zone/tcp/4430/wss"
    ]
  },
  "Daemon": {
    "PprofPort": -1
  }
}
marten-seemann commented 2 years ago

js-ipfs now returns a very unexpected error: Uncaught (in promise) Error: no protocol with name: "'dns'". Must have a valid family name: "{ip4, ip6, dns4, dns6}".

Looks like JS expects an IP address, not a domain name. The domain name is needed though to allow the browser to verify the certificate. @achingbrain, this seems like bug in js-libp2p, doesn't it?

lidel commented 2 years ago

Related /dns/ issues:

lidel commented 2 years ago

@TheDiscordian given that Kubo 0.14 has the updated go-libp2p already, one should be able to connect to relay via /dns just fine:

$ ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
connect 12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A success

Can you confirm things work on your end, and if so, close this issue?

lidel commented 2 years ago

@TheDiscordian after JS part is shipped in JS-IPFS (https://github.com/ipfs/js-ipfs/pull/4178), please make sure you update to libp2p-relay-daemon v0.2.0 before you re-test things end-to-end. This way, we also confirm the latest go-libp2p works fine :crossed_fingers:

lidel commented 2 years ago
TheDiscordian commented 1 year ago

@lidel @achingbrain this appears to be unresolved for js-ipfs, any theories?

Repo used: https://github.com/TheDiscordian/go-libp2p-relay-daemon

Code:

<!DOCTYPE html>
<html lang="en">
<head><title>js-ipfs minimal relay test</title>

<script src="https://cdn.jsdelivr.net/npm/ipfs-core@0.16.1/dist/index.min.js"></script>

<script>
var ipfs;
async function main() {
    ipfs = await window.IpfsCore.create();
    await ipfs.swarm.connect("/dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A");
}
main()
</script>
</head>
</html>

I can confirm this works fine /w Kubo. Error is the same as before /w js-ipfs, Chrome:

Screen Shot 2022-10-04 at 1 00 34 PM

Firefox:

Screen Shot 2022-10-04 at 1 01 29 PM

2color commented 1 year ago

I can confirm this works fine /w Kubo. Error is the same as before /w js-ipfs, Chrome:

So both Kubo and js-ipfs daemons are able to connect to /dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A But js-ipfs in the browser fails to connect?

TheDiscordian commented 1 year ago

@2color I haven't actually tested the js-ipfs daemon, only the browser. However yes, Kubo definitely works fine, I can connect and see the node in my list of peers.

lidel commented 1 year ago

jsipfs daemon fails too:

$ ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
connect 12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A success

$  jsipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
(node:16151) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
fetch failed
lidel commented 1 year ago

@achingbrain I wonder if JS side of things needs to implement the /sni/ fix from https://github.com/libp2p/go-libp2p/pull/1719 ?

marten-seemann commented 1 year ago

@achingbrain I wonder if JS side of things needs to implement the /sni/ fix from https://github.com/libp2p/go-libp2p/pull/1719 ?

It's only used internally, although one could announce addresses containing /sni.

thibmeu commented 1 year ago

kubo fails to contact a websocket server behind a reverse proxy, should it be a ws or wss. From my understanding, it comes from the following part in go-libp2p

// 1. initial addr is /dns/example.com/tcp/443/ws
resolvedAddrs, err := resolver.Resolve(ctx, a)
// 2. resolvedAddrs is /dns/example.com/tcp/443/ws
...
resolved, err := s.resolveAddrs(ctx, peer.AddrInfo{...})
// 3. resolved is /ip4/1.2.3.4/tcp/443/ws
// the information about the DNS host is lost, preventing a reverse proxy from directing the HTTP connection to the right host

The above is similar, but recoverable (not recovered) with wss. WSS adds /tls/sni/example.com/ws, which could be used by the websocket transport when dialing the resolved IP.

I think the multiaddr resolution is the component at fault here. /dns/example.com/tcp/443/ws is not equivalent to /ip4/1.2.3.4/tcp/443/ws. The IP is correct, but it's missing the domain name. websocket transport dial would be able to use an IP resolved by the Swarm resolver, but doesn't know the domain name to pass to the reverse proxy.

To address the issue (in go-libp2p), we could consider the following:

  1. add an sni component in the multiaddr resolved by websocket transport, even for non wss connection. Then, use this multiaddr component as the host for the connection, while NetDial dials the IP resolved by swarm
  2. add a dnsname component in the swarm resolved multiaddr. This would be a new multiaddr component. This dnsname should be ignored by other transports, but could be leveraged should they need to use the domain name.

I would be glad to contribute to an implementation. This caused issue on a server of mine, which cannot swarm connect with vanilla kubo. The fix I have is to use the sni component as Host (here), but this bypasses Swarm resolution.

BigLep commented 1 year ago
  • [ ] After we confirm this is fixed, we need to add smoke/regression test that dialls /dns/.../wss from js-ipfs and Kubo (tbd if this should live in their respective repos, or in https://github.com/ipfs/interop)

@lidel : I wish I knew the answer myself, but would you please be able to outline what tests are needed where so we don't regress on this in the future? I am then game to engage with the teams to make sure we harder ourselves here. Thanks.

p-shahi commented 1 year ago

go-libp2p v0.23.3 has been released https://github.com/libp2p/go-libp2p/releases/tag/v0.23.3 which incorporates @thibmeu 's fix for the missing HTTP Host header in /wss