--disable-api-security does not disable API security

[mtlynch]

The change in behavior of the --disable-api-security introduced in v1.0.1 is unintuitive and breaks user scenarios.

In 1.0.1, if the user passes --disable-api-security siad requires the user to add an API password. This is confusing because this does not seem to honor the user's request to disable the security measures on the API port; it just applies different security requirements.

This requirement breaks scenarios like Sia via Docker. It is not possible for anything outside of the Docker container to communicate with the container's localhost interface. Even if the user tries to add an API password, it's klunky to script because siad expects the password to be entered by an interactive user rather than as part of a script.

[lukechampine]

good catch. In the meantime, you can listen on localhost and forward connections to the daemon. (I believe nginx can do this; there are probably more lightweight alternatives)

[DavidVorick]

hmm.

This needs to be balanced against the fact that we have a lot of miners who will happily run the --disable-api-security command without properly securing their network and then lose all their coins. On one hand they kind of deserve it but on the other hand we need to protect them.

I'm not sure the best UX way to approach this, maybe @VoidingWarranties has some ideas? Goals:

• protect miners/hosts with multi-machine setups
• don't make life difficult for people using scripts and docker and whatever else

[DavidVorick]

I think this falls into a similar bucket as people who don't write down their passwords. No matter how obvious you make the warnings, people are happy to click through and miss critical things. We need a better way to make people inherently secure.

[VoidingWarranties]

API authentication was originally intended to protect against user-space malware on your local machine. As such, API auth uses HTTP basic authentication which sends passwords in plaintext. This is reasonably safe to do on the loopback interface, but is susceptible to MITM on other network interfaces.

Enabling authentication prevents against someone port scanning your machine and stealing your coins, but it also makes you vulnerable to a MITM attack if you make API requests from a compromised network.

An alternative solution to the docker annoyance is to allow the API password to be specified on the command line in a non-interactive way. Bitcoin allows you to do this with the -rpcauth flag which lets you specify the password's salt and hash. It wouldn't be difficult to do something similar here.

[VoidingWarranties]

The --disable-api-security flag is a misnomer if we require API authentication to be enabled. It is not the best UX, but it seems to have reduced the number of vulnerable nodes.

[mtlynch]

The 1.0.0 setup was secure by default, which is the right thing to do. The 1.0.1 behavior tries to make it "secure" no matter what the user wants, which I don't think is the right thing to do.

If the concern is that the users will just do whatever is easiest at the expense of security, they'll probably also just choose a weak API password. "a" as a password currently works.

Requiring Docker users to type a password on several different siac commands is a pretty big drag. Why punish Docker users who do the right thing in order to serve users who use non-default settings to create an insecure setup?

[VoidingWarranties]

Makes sense.

We originally planned on making API auth enabled by default, but I don't think we ever did. We could enable it by default and have the --disable-api-security flag disable it. Or we could have the --disable-api-security flag only enable non-localhost binding, and the users would still have to specify something like --authenticate-api false or maybe --disable-api-authentication. The latter seems like a safer option but maybe it's just more cumbersome. What do you think @mtlynch?

If you still want to enable API auth but don't want to type the password in siac interactively you can also specify the password like so --addr :<password>@localhost:9980.

[DavidVorick]

If the concern is that the users will just do whatever is easiest at the expense of security, they'll probably also just choose a weak API password. "a" as a password currently works.

At least choosing the password 'a' is better than not having a password at all, it filters out the branch of criminal that didn't bother to do a basic dictionary attack on each autheticated api.

We only use basic http auth, which is a pretty terrible thing on the whole, if you can sniff a connection between a legit client and a legit server you can get the password to the server, there's no encryption. So, simply using authentication definitely doesn't count as secure, and security should still be considered 'disabled` if you are off localhost, even if you are using a password.

This change was inspired by a week of reports where numerous miners (correlating to the release of mining pools) were complaining about stolen funds. Most of these miners had exposed their apis with no authentication at all. Secure by default was not enough because they were definitely getting themsevles pwned, and I can't take a hard line when 10+ users are having large amounts of funds stolen. We needed to make a change in the more protective direction.

At some point you have to draw the line and say 'we refuse to protect users who are X dumb', but we need to make sure that those users don't form a significant part of the userbase.

I think the right thing to do is to make it easier for docker users to deal with the password. If typing it into the terminal is too much, perhaps an optional command line arg is more acceptable?

I will not be comfortable changing the name of --disable-api-security until we've got something superior to basic http auth, but that doesn't seem to be in the near future.

[mtlynch]

At least choosing the password 'a' is better than not having a password at all, it filters out the branch of criminal that didn't bother to do a basic dictionary attack on each autheticated api.

For the password to be a useful control, we have to assume that the most sophisticated attacker is someone who knows how to port scan and rob Sia wallets, but doesn't know how to perform a dictionary attack. Until the number of users running siad is in the tens of thousands, a single attacker performing dictionary attacks can feasibly rob every single Sia user with a weak password. In other words, if we stop ten bad attackers by enforcing password requirements, but fail to stop one attacker capable of dictionary attacks, we didn't provide meaningful protection because that one attacker can compromise every Sia user.

This change was inspired by a week of reports where numerous miners (correlating to the release of mining pools) were complaining about stolen funds. Most of these miners had exposed their apis with no authentication at all. Secure by default was not enough because they were definitely getting themsevles pwned, and I can't take a hard line when 10+ users are having large amounts of funds stolen. We needed to make a change in the more protective direction.

Can you go into more detail about this attack? If you're pool mining, you don't need to run siad at all, so how could this happen? Even if they're running siad, what network are they on where there's just no firewall whatsoever? If I wanted to open up the siad API port to the Internet, I'd have to take several deliberate steps to do that, so I'm unclear how people are doing this accidentally.

FWIW, I just port scanned all the Sia hosts and none of them have port 9980 open to the Internet.

$  ./siac hostdb | grep -oP '[^\s]+:' | grep -oP [^:]+ > sia-hosts.txt
$ nmap -iL sia-hosts.txt -vvv -oA nmap-results -p 9980-9982 -Pn
$ grep 9982/open nmap-results.gnmap  | wc -l
65
$ grep 9980/open nmap-results.gnmap  | wc -l
0

[mtlynch]

Thanks for the responses @VoidingWarranties! My replies inline.

API authentication was originally intended to protect against user-space malware on your local machine. As such, API auth uses HTTP basic authentication which sends passwords in plaintext. This is reasonably safe to do on the loopback interface, but is susceptible to MITM on other network interfaces.

Sia should not attempt to defend against this scenario. If an attacker has code execution on the user's machine, it's game over. If there's an API password, they can sniff keystrokes or network traffic and very easily recover the password.

We originally planned on making API auth enabled by default, but I don't think we ever did. We could enable it by default and have the --disable-api-security flag disable it.

Requiring passwords by default seems unnecessary if the default is to listen on localhost. If we're listening on localhost, the password doesn't improve security because the attacker would have to already have code execution on the system if they're talking to localhost (assuming user hasn't set up strange port forwarding rules).

Or we could have the --disable-api-security flag only enable non-localhost binding, and the users would still have to specify something like --authenticate-api false or maybe --disable-api-authentication.

Do you have any particular scenarios in mind where we would want --authenticate-api false? For power users, it's, as you say, another cumbersome flag. For the confused user who's blindly adding --disable-api-security, presumably they'd just as blindly add --authenticate-api false, so we'd be in the same boat.

If you still want to enable API auth but don't want to type the password in siac interactively you can also specify the password like so --addr :@localhost:9980.

That's nicer, but still not ideal. To me, the whole forced authentication feels like an imposition to power users that only superficially protects confused users who use insecure, non-default settings. So reducing the burden to power users is good, but I'm not clear on what we realistically win by burdening them in this way at all.

[DavidVorick]

Ok, coming back to this in hopes of finding some closure.

FWIW, I just port scanned all the Sia hosts and none of them have port 9980 open to the Internet.

Most of the miners being compromised weren't running hosts. I'm guessing the attacker had some code to crawl the full list of nodes on the network (there's a nice file you keep locally with hundreds of entries) to scan ports and find vulnerable miners.

Sia should not attempt to defend against this scenario. If an attacker has code execution on the user's machine, it's game over. If there's an API password, they can sniff keystrokes or network traffic and very easily recover the password.

Userspace malware is not able to sniff keystrokes, nor is it able to sniff network traffic. You need root access to do those things. Malware has pretty limited abilities in userspace, we can use that to our advantage. They do have full access to the filesystem, which means they can do some nasty things with the consensus set, but that's a highly sophisticated attack at that point.

That's nicer, but still not ideal. To me, the whole forced authentication feels like an imposition to power users that only superficially protects confused users who use insecure, non-default settings. So reducing the burden to power users is good, but I'm not clear on what we realistically win by burdening them in this way at all.

If you are a power user, it's not that big of a burden to set up a password, and then specify the password as mentioned above. Normally I would side with you, --disable-api-security should be enough but we were still seeing users fall victim because they were just blindly adding that flag. Now that we've required both, I haven't seen any reports of vulnerabilities.

---

My favorite solution to all of this is to require the API to be served over localhost, and then have a guide for using standard tools (ssh, etc.) for talking to the API from another machine. If you are using an ssh server/client setup, or some equivalently secure setup, we don't have to worry about doing all of the security ourselves. Which is good, because security is hard and none of us are experts in this particular type of security.

I think we should consider the solution we have right now 'temporary', with the intention of blocking off exported API access in the near future. Exporting your API is a huge risk, even if it's on a local network you are trusting that you've configured the firewall completely correctly. Anyone who can sniff traffic can compromise you, and anyone able to guess the password can also compromise you.

[mtlynch]

Most of the miners being compromised weren't running hosts. I'm guessing the attacker had some code to crawl the full list of nodes on the network (there's a nice file you keep locally with hundreds of entries) to scan ports and find vulnerable miners.

Ah, gotcha. I assume you're talking about /gateway/nodes.json? I just dumped the IPs from this and scanned them. Of 1107 nodes, 22 of them are listening on 9980.

Do we know why these users are opening up port 9980 in the first place?

It feels like we're attempting to solve this problem at the wrong level. If Sia users are running into some problem and they're solving it (or at least they think they are) by opening up siad to the world, we should figure out what it is they can't do with the default, secure configuration and fix that.

Userspace malware is not able to sniff keystrokes, nor is it able to sniff network traffic. You need root access to do those things. Malware has pretty limited abilities in userspace, we can use that to our advantage. They do have full access to the filesystem, which means they can do some nasty things with the consensus set, but that's a highly sophisticated attack at that point.

I think defending against userspace malware is a waste of the Sia dev team's time. I think the right solution to contain malware is separate user contexts and containerization, not app-level protections. I can't think of any major software that tries to defend against userspace malware like this except for security software or OSes themselves.

My favorite solution to all of this is to require the API to be served over localhost, and then have a guide for using standard tools (ssh, etc.) for talking to the API from another machine. If you are using an ssh server/client setup, or some equivalently secure setup, we don't have to worry about doing all of the security ourselves. Which is good, because security is hard and none of us are experts in this particular type of security.

I think if the Sia dev team feels nervous about this aspect of security, the smartest thing to do is to model based on what other backend systems do. bitcoind, IIUC will allow you to do listen for RPCs without a password (and they also allow configuration via a config file, which is less painful than relying on interactive prompts).

But even outside of cryptocurrency, if we look at things like backend databases (MySQL, MongoDB, Redis), none of those will insist on only accepting commands over localhost. They also won't fight the user if the user elects to listen on all network interfaces.

One big reason is that it would get in the way of good software architecture. As people begin developing apps on top of Sia, it should be designed such that deployment is flexible and accommodating to service isolation. If we think about something like MegaUpload, but using Sia as a backend, an obvious architecture is probably something like:

            internet             |            backend
[ end user ] --> [ web server ] -|-> [ application server ] --> [ siad ]

Where each box is a separate machine or container. But if we're restrictive about the way that siad listens for RPCs, either the application server and siad have to be smushed into the same machine or the system has to be architected such that the app server talks to siad over SSH, which would be a huge pain.

[mtlynch]

Malware has pretty limited abilities in userspace, we can use that to our advantage. They do have full access to the filesystem, which means they can do some nasty things with the consensus set, but that's a highly sophisticated attack at that point.

I've been thinking about this a bit more because I really feel like trying to defend against malware running on the same host is a losing battle. If I can get code execution on a machine running a Sia node, I can overwrite the siac binary with my own malicious binary that acts like siac but empties the victim's wallet as soon as it is unlocked.

[DavidVorick]

Do we know why these users are opening up port 9980 in the first place?

It feels like we're attempting to solve this problem at the wrong level.

Quite possibly because they are being instructed to by a mining pool. Either that, or they do not know any other way to connect a large number of mining computers to a single master siad instance.

I agree that resolving their use-cases is preferable to a heavy-handed ban on bad configurations. But I am not sure how broad the use cases are.

I think defending against userspace malware is a waste of the Sia dev team's time.

I can overwrite the siac binary with my own malicious binary that acts like siac but empties the victim's wallet as soon as it is unlocked.

Yes, there's not much we can do about that. Well, some operating systems I believe support binary-signing, which means you would not be able to do that. But it's not standard and we certainly aren't using it at the moment. And even if we were, the database file would need to be signed as well, because you can do similarly nefarious things by modifying the database file.

BUT, simply by preventing password files from ever being written to disk we can stop a broad range of script-kiddie malware, which I think is how most coins end up being stolen. It takes a much more sophisticated hacker to insert a replacement siac or to modify the database, so much so that the attacker has to be using outright sia-specific malware to execute any attacks.

General userspace malware is very common, and I consider it to be a huge gain that we protect against general userspace malware even if we are unable to protect against Sia-specific userspace malware.

---

I'm increasingly okay with removing the password requirement, but I still hesitate because I know that reports of stolen coins dramatically decreased as soon as we added the password requirement. We have pretty strong evidence that the password requirement has actively saved users money, to the tune of millions of siacoins.

All of the vulnerable users are already using passwords now, so dropping it may mean that they will continue to use passwords. I think we should add a warning in red in the output though - "This setup puts your coins at risk of being stolen - adding a password with '-p mypassword' is strongly recommended." - to catch any users who are ignorantly exposing their API to the public.

We should probably have the warning in red even with the password, given that it's basic HTTP auth and no good against anyone who can sniff the traffic.

[mtlynch]

Quite possibly because they are being instructed to by a mining pool.

I'm not sure I follow. Why would a mining pool instruct anyone to do that? My understanding is that anyone pool mining doesn't need to run siad at all.

General userspace malware is very common, and I consider it to be a huge gain that we protect against general userspace malware even if we are unable to protect against Sia-specific userspace malware.

I'm confused by this. Isn't Sia-specific malware all we're talking about? Any malware that puts Sia at risk has to be Sia-aware. There isn't any generic malware that can figure out what Sia is without having ever been programmed specifically to attack Sia.

I know that reports of stolen coins dramatically decreased as soon as we added the password requirement. We have pretty strong evidence that the password requirement has actively saved users money, to the tune of millions of siacoins.

Are any of these reports public? I tried searching the sia.tech forums and couldn't find reports of Sia users having coins stolen.

[DavidVorick]

I'm not sure I follow. Why would a mining pool instruct anyone to do that? My understanding is that anyone pool mining doesn't need to run siad at all.

I am not sure either. My assumption was that some how-to guide somewhere had some bad advice, just based on the types of reports we were getting. I should have investigated further.

I'm confused by this. Isn't Sia-specific malware all we're talking about? Any malware that puts Sia at risk has to be Sia-aware. There isn't any generic malware that can figure out what Sia is without having ever been programmed specifically to attack Sia.

There's a lot of malware that has generic tools to sweep up anything that looks like a wallet file or a password file. Considering the sia wallet is called 'wallet.json', it's not too much of a stretch to see it getting picked up. The wallet used to be called 'wallet.dat', which is the name of the bitcoin file, which is much more likely to get swept up.

I guess I am not an expert on it, but my understanding has always been that most malware is run by people other than the writers of the malware.

Other than some active network-based attacks (which did seem to be Sia-specific), I don't think I've seen any reports congruent with wallet files being stolen. I do think that this is in-part due to the fact that we refuse to save the user password to disk. Hard to be certain though, as we don't have any sort of A/B testing in our releases.

Are any of these reports public? I tried searching the sia.tech forums and couldn't find reports of Sia users having coins stolen.

Some were publicly discussed in the slack, many in the #mining channel. Several were just private messages directly to me. I don't think that any made it all the way to the forum.

[mtlynch]

I think you might be confusing this discussion with #1403. We're not talking about saving a password to file here, but rather requiring a password on the siad API.

@VoidingWarranties said this:

API authentication was originally intended to protect against user-space malware on your local machine. As such, API auth uses HTTP basic authentication which sends passwords in plaintext. This is reasonably safe to do on the loopback interface, but is susceptible to MITM on other network interfaces.

I'm saying that for malware to do this, it has to be programmed specifically to target Sia. If the attacker is going to the trouble of writing Sia-aware malware, they might as well replace the binary with an evil siac binary that drains the wallet as soon as the user enters their wallet password.

So with this in mind, I don't think an API password for siad offers any protection against userspace malware.

I understand what you're saying in that it offers some degree of protection to users who expose 9980 to the public Internet. If you require a password to listen outside localhost, at least now the attacker has to do the work of brute forcing the password, but I see this as pretty trivial for a Sia-focused attacker. So it effectively becomes a speed bump to attackers, but a real problem for legitimate users who want a painless way to deploy siad in things like Docker containers or isolated VMs.

[mtlynch]

Closing per @DavidVorick's comment in #1387.

I've tested a workaround that uses socat to port forward to 9980:

socat tcp-listen:8000,reuseaddr,fork tcp:localhost:9980 & ./siad

siad is happy because it's only listening on localhost:9980, but we can still access it from 8000 as a non-localhost port.

More details: http://mtlynch.io/sia-via-docker/

[lukechampine]

Thanks for posting a workaround. I hope that we can revisit this at some point and arrive at a solution that doesn't require a workaround.