IPv4 NAT table overflows

[the8472]

For IPv4 I'm behind several layers of NAT while ipv6 is unfiltered.

DHT traffic is poison for NAT connection tracking tables. So my question is whether it's possible to configure the daemon in a way that avoids causing ipv4 traffic.

E.g. by binding the DHT to v6-only while allowing mixed v4 and v6 swarm connections.

[whyrusleeping]

I think the best you could do is to use only ipv6 for your node. Leaking knowledge about the type of transport per service breaks the abstractions we're working to build.

[the8472]

That seems reasonable and I'll do that for now.

But even if i put it into "kinda-v6-only" mode there could still be some room for opportunistic v4 connections, no?

[the8472]

also, how do I actually restrict it to ipv6?

even with swarm: ["/ip6/::/tcp/4001"] there still seem to be a of outgoing v4 connections.

[the8472]

Another point, an interative DHT over TCP is even worse for stateful NATs due to different conntrack timeouts for TCP and UDP.

Lookups visit many different IP addresses with short-lived exchanges. When they're TCP they clog the tables much long.

You mentioned on IRC that 0.4.0 will have a UDP-based DHT. I think an option to make it UDP-only on v4 may be an alternative to making it v6-only.

[jbenet]

E.g. by binding the DHT to v6-only while allowing mixed v4 and v6 swarm connections.

Something that's related to this is that we do plan to eventually pick between connections using stats (bw, latency estimators, networks, etc). if we come up with nice, general abstractions for expressing constraints based on the qualities or capabilities, we could see about it. One related thing we would like to be able to do is latency/bw QoS -- i.e. prefer low-latency conns for some services or even requests, and bump the less latency-dependent things to other (possibly higher bw) conns.

this is not easy, and not our focus any time soon. I suspect the libp2p effort will bring a lot of people interested in this, so we could see about being able to upgrade the libp2p net abstractions to cleanly express this sort of thing.

Another point, an interative DHT over TCP is even worse for stateful NATs due to different conntrack timeouts for TCP and UDP.

Yes indeed. TCP is mostly just to simplify dev. We just landed uTP support in 0.4.0. this should help. quic will help down the road too. We can see about adding curvecp and other transports to libp2p too.

You mentioned on IRC that 0.4.0 will have a UDP-based DHT.

that's not right-- we'll have utp/udp support, but that's still connection streams, not a packet-based DHT.

I think an option to make it UDP-only on v4 may be an alternative to making it v6-only.

you can do this in v0.4.0-dev, with a config like:

> ipfs config Addresses.Swarm
[
  "/ip6/::/tcp/4001",
  "/ip6/::/utp/4002",
  "/ip4/0.0.0.0/utp/4002",
]

[the8472]

that's not right-- we'll have utp/udp support, but that's still connection streams, not a packet-based DHT.

Ah, I'm not sure if that will be good enough. connection tracking may treat a 2-way exchange differently from >=3-way connectivity. I.e. bump timeouts once the 3rd packet is seen.

you can do this in v0.4.0-dev, with a config like:

well, will this actually work? as I mentioned above i've tried with the current 3.x version and there the Addresses.Swarm setting fails in some way. I think it only restricts the listening socket, but it does not prevent outgoing connections from other source addresses.

[jbenet]

I think it only restricts the listening socket, but it does not prevent outgoing connections from other source addresses.

oh interesting. you're right. we should restrict both outgoing and incoming. hmmmm :/

@whyrusleeping maybe swarm filters should allow specifying the full transports too

[badb]

Are there any chances to put connection limits higher on priority list? Starting from 0.4.0, IPFS has started to create thousands of connections per daemon. Just two instances overflow our router's NAT table, effectively cutting down internet access for everyone in the company. We downgraded to 0.3.11, which keeps a reasonable number of connections. Changing Swarm address transport protocol to UDP/uTP also doesn't help much - the network needs less time to recover, but overflow still occurs.

[whyrusleeping]

@badb hrm... the outgoing connections shouldnt overflow your routers NAT table if our TCP reuseport and NAT traversal code is working properly. Now that i re-read this issue I think its more of a problem than I originally assumed, its likely we've got something going haywire with the upnp reuqests, or that reuseport is tripping over itself.

[the8472]

That depends on the NAT implementation.

Not all stateful NAT engines assign only one table entry per EIM (endpoint independent mapping). For example iptable's conntrack always tracks individual pairs of socket addresses even if the abstract mapping is only based on the local socket address. This even applies when you use fixed snat/dnat rules and even to UDP.

That means the the number of table entries is still related to the number of pseudo-connections and not the endpoint mappings.

There used to be some xtables extension modules that supported stateless, static nat rules but for some reason those commits were reverted. So effectively any and all iptables-based solutions, barring custom patching, will either require large tables or very short timeouts otherwise they will experience table overflows.

Based on some blackbox-testing I have done against my ISP's DS-Lite carrier-grade NAT their behavior is comparable to that of iptables.

In other words, reuseport and fixed UDP endpoints may aid nat traversal, but they do not necessarily prevent nat table overflows.

[whyrusleeping]

@the8472 awesome information, thanks! In that case, the only real solution is "stop opening so many connections" right?

[the8472]

In first approximation: Yes

If you look more closely it's more complicated. The number of currently open connections only gives you a lower bound of possibly present nat table entries because those entries can linger after a connection has been closed or a connection attempt has failed. Additionally incoming connections or connection attempts from other nodes can also spawn nat table entries.

So a conservative approach is to both limit the number of connection attempts per unit of time and the number of concurrently open connections. additionally changing ports every now and then can help to not appear "too stable" for other nodes to put a limit on how many of them will try to connect.

Since this generally affects IPv4 more severely than IPv6 (it's usually easier to put the latter into a stateless configuration) those limits should be configurable separately. A v6-first, v4-fallback approach taken at every step can help further. I.e. the opposite of the happy eyeballs approach.

[Kubuxu]

There is no real NAT66 (only NPT66 which is stateless) so it shouldn't be a problem for IPv6.

[the8472]

I have focused on NAT, but state table overflows exist in stateful firewalls and thus IPv6 too.

It's just a lesser issue with v6 because stateful firewalls can be turned off. Stateful nats on the other hand usually can not.

[Stebalien]

Duplicate of https://github.com/ipfs/go-ipfs/issues/3320.