IPFS is doing port scanning, without indicating that in logs

burdakovd commented 6 years ago

Version information:

go-ipfs version: 0.4.15- Repo version: 6 System version: amd64/linux Golang version: go1.9.7

Type:

Bug

Description:

IPFS randomly connects to addresses in a way that resembles port scanning, without any indication of what is happening in the logs (level=debug).

A few days after installing IPFS daemon on my public server, I've got email from provider (AWS) with abuse report, indicating that sombeody received a port scan from my address (700 TCP ports scanned in total).

To be safer and not get banned, I've done the following steps:

Disable all NAT traversal, MDNS, and other "magic" technologies in IPFS server (as based on the searches, they may be responsible for things similar to port scanning)
To ensure I don't get blamed for port scanning, configured firewall to allow outgoing connections to only pot 4001
Configured IPFS to advertise only my public IP address (and not e.g. 127.0.0.1, or internal address of Docker container), to ensure none of the NAT "piercing" logic is needed.
Added logging on AWS to monitor all traffic going to and from machine
Added debug level logging on ipfs daemon

This is my full config: https://ipfs.io/ipfs/QmQUWSFqV9bDckgpzBaKBBZsPSwHCxVAjKRx31TD5zuLVV/ This is how I generate the config (start script of my Docker image): https://raw.githubusercontent.com/burdakovd/dapps.earth/master/ipfs/start.sh

Now, I started looking into rejected outgoing connections, and indeed I see behavior resembling port scan, despite all my settings. Luckily this time, my local firewall prevented another abuse report.

Here are logs indicating my node (internal IP is show in logs) connecting to 54.186.184.82. First connection is OK, and then 10 minutes later it starts connecting to random ports of 54.186.184.82.

Schema: [version, account, eni, source, destination, srcport, destport, protocol, packets, bytes, windowstart, windowend, action, flowlogstatus] ACCEPT/REJECT means just whether the connection was accepted/rejected by my firewall, not necessarily means that the connection successfully went through, as it could've been rejected on the other side.

In the meantime, in ipfs logs (debug level of logging enabled), there is no mention of 54.186.184.82 whatsoever!

(note: 01:42:00 is the moment when I have restarted my node, after having updated log level to "debug")

Furthermore, I see that in third line of network logs 54.186.184.82 was attempting to connect to me at port 1024, despite me advertising only 4001.

bonedaddy commented 6 years ago

enable the server profile during your init statement. ipfs init --profile=server

also I don't think it's doing port scanning I suspect the bitswap traffic might be detected as port scanning.

burdakovd commented 6 years ago

@postables , I did enable server profile.

I did it with ipfs config profile apply server though instead of ipfs init --profile=server, I don't think it should make difference. I've included the full resulting config in the issue description.

Is it expected that bitswap would attempt to connect to multiple ports of a peer (without indicating that in logs)?

To start with, I'd be surprised if a peer advertises themselves listening on 8 different ports, why would they do so? Furthermore, there was no mentions of peer IP address in debug-level logs whatsoever.

I'm not that familiar with internals of how bitswap works, but is it normal for it to attempt to connect to 700 different TCP ports of a given IP address (which is what I've received in initial abuse report)?

burdakovd commented 6 years ago

Another example:

In this case we've had an incoming connection from 171.214.222.88. After some period of exchange, it appears disconnect happened.

After that my node just started dialing random ports of 171.214.222.88, until it finally reestablished connection at 171.214.222.88:59512.

This time there are logs from ipfs for this session (intersting part at 16:54:51): https://ipfs.io/ipfs/QmRKhMz6vtftti5PFaCqQAGESBNjBWmEqaE5TCbXfgpn88/

Particularly worrying part is below:

It seems in this situation, after connection has been lost (maybe remote node was restarting, or temporarily went offline), we bombarded 171.214.222.88 with attempts to connect on many differenr ports. Those attempts were unsuccessful, and the connection got recovered after that node finally connected to us.

It appears that node is behind NAT, as this is what they are advertising: [/ip4/192.168.100.29/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/171.214.222.88/tcp/59606 /ip4/171.214.222.88/tcp/59512]. It advertises a bunch of local IP addresses, and a public IP with an ephemeral port generated by NAT.

For some reason I'm also getting TONS of ports of 171.214.222.88 from PEERS CLOSER messages, e.g., and this is how my node got an idea to attempt to connect to many ports of 171.214.222.88:

[0;37m15:34:07.717 [36mDEBUG [0;34m dht: [0mPEERS CLOSER -- worker for: <peer.ID fEwokY> added <peer.ID Z7jpNP> ([/ip4/171.214.222.88/tcp/62272 /ip4/171.214.222.88/tcp/60469 /ip4/171.214.222.88/tcp/62836 /ip6/::1/tcp/4001 /ip4/171.214.222.88/tcp/56383 /ip4/171.214.222.88/tcp/53663 /ip4/171.214.222.88/tcp/57890 /ip4/171.214.222.88/tcp/63478 /ip4/171.214.222.88/tcp/54038 /ip4/171.214.222.88/tcp/15271 /ip4/171.214.222.88/tcp/3811 /ip4/171.214.222.88/tcp/61617 /ip4/171.214.222.88/tcp/9751 /ip4/192.168.100.29/tcp/4001 /ip4/171.214.222.88/tcp/62421 /ip4/171.214.222.88/tcp/53483 /ip4/171.214.222.88/tcp/59495 /ip4/171.214.222.88/tcp/11817 /ip4/171.214.222.88/tcp/62354 /ip4/171.214.222.88/tcp/60000 /ip4/171.214.222.88/tcp/54804 /ip4/171.214.222.88/tcp/49942 /ip4/171.214.222.88/tcp/44293 /ip4/171.214.222.88/tcp/57158 /ip4/171.214.222.88/tcp/63682 /ip4/171.214.222.88/tcp/54838 /ip4/171.214.222.88/tcp/19213 /ip4/171.214.222.88/tcp/52111 /ip4/171.214.222.88/tcp/49093 /ip4/171.214.222.88/tcp/60562 /ip4/171.214.222.88/tcp/58653 /ip4/171.214.222.88/tcp/60472 /ip4/171.214.222.88/tcp/59542 /ip4/171.214.222.88/tcp/11749 /ip4/171.214.222.88/tcp/27011 /ip4/171.214.222.88/tcp/2191 /ip4/171.214.222.88/tcp/52826 /ip4/171.214.222.88/tcp/59978 /ip4/171.214.222.88/tcp/64238 /ip4/171.214.222.88/tcp/34921 /ip4/171.214.222.88/tcp/48367 /ip4/171.214.222.88/tcp/64888 /ip4/171.214.222.88/tcp/63596 /ip4/171.214.222.88/tcp/50054 /ip4/171.214.222.88/tcp/59972 /ip4/171.214.222.88/tcp/52387 /ip4/171.214.222.88/tcp/8553 /ip4/171.214.222.88/tcp/53991 /ip4/171.214.222.88/tcp/56400 /ip4/171.214.222.88/tcp/36687 /ip4/171.214.222.88/tcp/5375 /ip4/171.214.222.88/tcp/50710 /ip4/171.214.222.88/tcp/35497 /ip4/171.214.222.88/tcp/61058 /ip4/171.214.222.88/tcp/50270 /ip4/171.214.222.88/tcp/61527 /ip4/171.214.222.88/tcp/54526 /ip4/171.214.222.88/tcp/53606 /ip4/171.214.222.88/tcp/64457 /ip4/171.214.222.88/tcp/57888 /ip4/171.214.222.88/tcp/57645 /ip4/171.214.222.88/tcp/57587 /ip4/171.214.222.88/tcp/48227 /ip4/171.214.222.88/tcp/2007 /ip4/171.214.222.88/tcp/56086 /ip4/171.214.222.88/tcp/51588 /ip4/171.214.222.88/tcp/6379 /ip4/171.214.222.88/tcp/62904 /ip4/171.214.222.88/tcp/22604 /ip4/171.214.222.88/tcp/60315 /ip4/171.214.222.88/tcp/63470 /ip4/171.214.222.88/tcp/30586 /ip4/171.214.222.88/tcp/19287 /ip4/171.214.222.88/tcp/19443 /ip4/171.214.222.88/tcp/19123 /ip4/171.214.222.88/tcp/9129 /ip4/171.214.222.88/tcp/45425 /ip4/171.214.222.88/tcp/31301 /ip4/171.214.222.88/tcp/48393 /ip4/171.214.222.88/tcp/61744 /ip4/171.214.222.88/tcp/59020 /ip4/171.214.222.88/tcp/14110 /ip4/171.214.222.88/tcp/16426 /ip4/171.214.222.88/tcp/57084 /ip4/171.214.222.88/tcp/57928 /ip4/171.214.222.88/tcp/33319 /ip4/171.214.222.88/tcp/51117 /ip4/171.214.222.88/tcp/50408 /ip4/171.214.222.88/tcp/62295 /ip4/171.214.222.88/tcp/15373 /ip4/171.214.222.88/tcp/52290 /ip4/171.214.222.88/tcp/55704 /ip4/171.214.222.88/tcp/10987 /ip4/171.214.222.88/tcp/54847 /ip4/171.214.222.88/tcp/64208 /ip4/171.214.222.88/tcp/26163 /ip4/171.214.222.88/tcp/53140 /ip4/171.214.222.88/tcp/59628 /ip4/171.214.222.88/tcp/65120 /ip4/171.214.222.88/tcp/50560 /ip4/171.214.222.88/tcp/53366 /ip4/171.214.222.88/tcp/52089 /ip4/171.214.222.88/tcp/25153 /ip4/127.0.0.1/tcp/4001 /ip4/171.214.222.88/tcp/58684 /ip4/171.214.222.88/tcp/58884 /ip4/171.214.222.88/tcp/63299 /ip4/171.214.222.88/tcp/58365 /ip4/171.214.222.88/tcp/51140 /ip4/171.214.222.88/tcp/41977 /ip4/171.214.222.88/tcp/60675 /ip4/171.214.222.88/tcp/2977 /ip4/171.214.222.88/tcp/28746 /ip4/171.214.222.88/tcp/54140 /ip4/171.214.222.88/tcp/54874 /ip4/171.214.222.88/tcp/9634 /ip4/171.214.222.88/tcp/60342 /ip4/171.214.222.88/tcp/19743 /ip4/171.214.222.88/tcp/58104 /ip4/171.214.222.88/tcp/54546 /ip4/171.214.222.88/tcp/4607 /ip4/171.214.222.88/tcp/64964 /ip4/171.214.222.88/tcp/59606 /ip4/171.214.222.88/tcp/42267 /ip4/171.214.222.88/tcp/63234 /ip4/171.214.222.88/tcp/38997 /ip4/171.214.222.88/tcp/58352 /ip4/171.214.222.88/tcp/62192 /ip4/171.214.222.88/tcp/45225 /ip4/171.214.222.88/tcp/57315 /ip4/171.214.222.88/tcp/37197 /ip4/171.214.222.88/tcp/65230 /ip4/171.214.222.88/tcp/61424

It would appear that this giant list of ephemeral ports is just because node behind 171.214.222.88 was connecting to many peers, and they recorded from which port the connection happened, and passed it to each other. I don't think in that case it it reasonable to pass those ports to each other or attempt to connect to them.

Why are we connecting to so many ports? Can I configure IPFS daemon to only connect to port 4001 (which would work for nodes with public IP)? If some other node does not have public IP, and is behind the NAT, I won't connect to them, but they could dial me any time.

bonedaddy commented 6 years ago

If you're node is advertising all locally connected interfaces then it doesn't appear that the server profile config change persisted as it is designed to stop this kind of advertisement

edit: it appears I misread your comment and that the node advertising all those addresses is a remote node

edit2: what I think might be happening is that the node you're dialing to on multiple ports is advertising content that your node needs so it's attempting to connect to to that peer on all known interfaces.

edit 3: the inverse could be true your node has content which that peer needs so that peer and your node are constantly trying to talk to each other

edit4: this issue may also be worth checking out https://github.com/ipfs/go-ipfs/issues/1226

edit5: this is another issue worth looking at https://github.com/ipfs/go-ipfs/issues/4343

burdakovd commented 6 years ago

edit2: what I think might be happening is that the node you're dialing to on multiple ports is advertising content that your node needs so it's attempting to connect to to that peer on all known interfaces.

edit 3: the inverse could be true your node has content which that peer needs so that peer and your node are constantly trying to talk to each other

My node doesn't need anything, and nobody can need stuff from it, since I'm not using it yet, it is a fresh new node. So this is purely a DHT traffic. I don't mind that, but I want to make sure that it only attempts to connect peers in reasonable way, and not by spamming them on dozens of ports.

I've read https://github.com/ipfs/go-ipfs/issues/1226 and https://github.com/ipfs/go-ipfs/issues/4343 and those are about node attempting to connect to local addresses. That is mitigated by server profile (which populates Swarm.AddrFilters), and isn't the case in my issue, as my node is attempting to connect to public addresses but on too many ports.

bonedaddy commented 6 years ago

You might be able to mitigate this routing the dhtclient routing mode. If that doesn't work then the only thing I can think of is to use iptables or something similar to block connecting to the offending node.

magik6k commented 6 years ago

The problem is likely that the node is advertising too many ports, like in https://github.com/ipfs/go-ipfs/issues/3780.

gallizoltan commented 6 years ago

Same issue here. I started to run IPFS daemon on AWS, and I've got an "Abuse Report" email about "activity which resembles scanning hosts on the internet for security vulnerabilities". I'm going to try ipfs init --profile=server now.

alexanderkjeldaas commented 6 years ago

Same here. Why is this not the default? Currently it's HIGHLY DANGEROUS (for business) to run IPFS.

burdakovd commented 6 years ago

To be clear, this particular issue is not about the following two things: 1) Why ipfs is not enabling server profiler by default 2) Why it is attempting to connect to local addresses (which has beend discussed in other issues before)

It is rather about why after enabling server profile, ipfs still attempts to connect to the same node (public IP) at dozens different ports, and how do we prevent this from triggering ISP abuse detection.

It might be a bug in the logic of how ports of that node are advertised and forwarded between peers (i.e. if node A is behind NAT and connects to dozens of peers, and all of them remember and forward the public host:port that they receieved connection from, at the end DHT will contain many records of the same public ip at different ports).

If we aren't able to identify why this is happening and fix it quickly, there are some remediations.

Remediation 1. On firewall allow outgoing connections to any port except 4001. This potentially hurts connectivity somewhat, but does help to prevent any activity that can resemble port scanning. This is what I've chosen.

Remediation 2. Change code of IPFS such that it only attempts to connect to new port of an IP address not more than once per day. Maybe also add limitation that it attempts to connect to an IP address not more than once per hour. Of course those limitations should be configurable. Once that is in place, even if some misbehaving node advertises thousands of closed ports on the same IP address, IPFS won't attempt to "scan" all of them in search of the one that is open. It wouldn't hurt connectivity with good peers, as they are likely advertising just one port.

alexanderkjeldaas commented 6 years ago

It might be a bug in the logic of how ports of that node are advertised and forwarded between peers (i.e. if node A is behind NAT and connects to dozens of peers, and all of them remember and forward the public host:port that they receieved connection from, at the end DT will contain many records of the same public ip at different ports).

Does IPFS use STUN server?

Stebalien commented 4 years ago

Closing in favor of the meta issue https://github.com/ipfs/go-ipfs/issues/6932.

ipfs / kubo