I've just wasted 2 days because of 2.0.0 new security (IoT purpose here)

Hi,

When i started my endeavors in the IoT world, Mosquitto was the MQTT broker that was advised roughly everywhere. It's lightweight, easy to use, no configuration needed and scales pretty well. Even sites like DigiKey still have Mosquitto as example: https://www.digikey.com/en/maker/projects/send-and-receive-messages-to-your-iot-devices-using-mqtt/39ed5690cc46473abe8904c8f960341f

Fast forward to a couple of days ago. I came home after being gone from a few weeks and had the very nasty surprise of my lights not working anymore. After a lot of debugging it started to look like the internet service provider (ISP) modem was to blame. As my light apparently didn't want to connect to WiFi from their modem but did want to connect to a tether with the same settings. In hindsight, this turned out to be a "false positive" as the lights in actuality were getting errors from trying to connect to MQTT which were errors when connected via the ISP modem (same network) and timeout on the tether (different network).

Still, when debugging even more in the firmware of the device itself, it started to look more and more like an ISP issue. As the wifi AP was giving me a response of 8 BSS (read https://wifiwiki.wordpress.com/2020/04/27/802-11v-wireless-network-management/) which also indicated that the router was kicking my lights out (pun intended).

This, in hindsight, turned out to be another false positive. I still don't get how that 8 response code works or how it's supposed to work, but it's apparently no issue for the wifi connection. The connection is just there and the device is reachable. (we're talking about an ESP8266 here)

Next up was inspecting my MQTT. My intent was just to see if some other devices that did still work was connected to the MQTT. I was again thrown off here because that device - which was supposed to use MQTT - was working fine (that turned out to be caused by something else entirely). Finally i decided to look at MQTT logs and noticed: Starting in local only mode. Connections will only be possible from clients running on this machine.

That looked like it could be an issue. As none of my ESP8266 devices has any network authentication for MQTT and are definitely not in the localhost of the pc that was running MQTT. Some more googling (and actually reading the 2.0.0 release blog) pointed me to the - finally - correct fix of this 2-day annoying debug hunt.

Thank you, Mosquitto, for wasting 2 days. That's not appreciated!

But... It's not your fault either. You did everything as i would expect a major breaking change to happen. It's a new major release. It has a blog post detailing what is changed and how to get things working. Really, you did it all just fine! It's just that the issue for me was, at first, not pointing to MQTT as being a problem.

Having said this though, your new default settings do make you more of a pain to setup for the IoT world. I get that for added security this measure was taken. But it comes at the cost of being slightly more tricky to setup. Is that really worth it? I mean, sure, exposing an MQTT network wide is very much an attack vector that is now closed by default. But then again, many articles promoting MQTT use it for network devices like myself. If one then is stupid enough to expose that very MQTT to the internet too then it's, in my book, a own fault kind of thing.

Lastly, please don't consider any of this as hard critique against Mosquitto or it's developers. It's meant to be read as a somewhat funny story about how i wasted a couple of days. It's also meant as a question to get some clarity on the security intentions. Is this MQTT broker still advisable to be used in a IoT setting?

Best regards and thank you for the nice MQTT broker :) Mark

I'm sure the intent of the change is just to prevent people from accidentally opening up their broker to the world when they're trying it out on a machine with a public IP address. For your scenario, it's now "minimal config" rather than "no config", but the config that matches the old default behaviour is pretty minimal:

listener 1883
allow_anonymous true

It's not a bad idea to also put a host name or IP address on the listener line to restrict access to that IP.

I'm also sure there's no intent to make mosquitto less useful in IoT environments. The project sponsor provides professional services in the IoT space.

It's not a bad idea to also put a host name or IP address on the listener line to restrict access to that IP.

I probably misunderstand you here. What benefit does it give for me to add my hostname there? If i'm reading the docs correctly then it means that only MQTT from that very same host are allowed in that case.

Say i have a network with 10 WIFI enabled devices all communicating via MQTT and all with their own unique IP within the network. Surely you're not suggesting to add all of those ip's in that config too? If you were, that would severely break any automated provisioning for devices that would use Mosquitto as MQTT message transport.

No, the config affects which interfaces mosquitto listens at, or to put it differently the address your clients connect to. If you don't specify an IP address or host name, it will listen at all IP addresses on the machine running mosquitto. If it has only one network interface, it doesn't make much difference, but if it has more than one interface, it might help avoid errors or bad exposures if you configure the server to use only the address your clients will connect to.

Suppose the machine running mosquitto has an ethernet port at 192.0.2.47 and a wifi adapter at 198.51.100.47, and the devices that need to talk to it are at 198.51.100.30 to 198.51.100.39. If you have this:

listener 1883
allow_anonymous true

your lights will work, but you run the risk of some rogue device on the 192.0.2 network connecting to that mosquitto broker by mistake. Maybe you have an incandescent bulb which can only communicate over ethernet and it will drive your electricity bill up or maybe it's a coffee maker which is supposed to connect to a different broker but was misconfigured and it will unexpectedly stop working one day when you disconnect the ethernet from this machine, thinking the coffee maker was getting its instructions from somewhere else. In the worst case, maybe 192.0.2 is a public network and some evil-doer could start flicking your lights on and off at random intervals, just to annoy you.

If you put this instead:

listener 1883 198.51.100.47
allow_anonymous true

you'll only be able to connect to mosquitto through IP 198.51.100.47, so assuming there are no routes in to the wifi network, only devices on that network will be able to connect to the broker. Plus your config file provides some indication of how you expect your network to be laid out. I'm not saying this is important or even calling it a "best practice", but I think it's not a bad idea.

One caveat: if you want to access the broker through some other IP address (including localhost), you would need to define multiple listeners, one for each of those interfaces, so your still-less-minimal config would be something like:

listener 1883 127.0.0.1
listener 1883 198.51.100.47
allow_anonymous true

Maybe I have a high tolerance for pain, but I prefer that to listening at all addresses.

your lights will work, but you run the risk of some rogue device on the 192.0.2 network connecting to that mosquitto broker by mistake. Maybe you have an incandescent bulb which can only communicate over ethernet and it will drive your electricity bill up or maybe it's a coffee maker which is supposed to connect to a different broker but was misconfigured and it will unexpectedly stop working one day when you disconnect the ethernet from this machine, thinking the coffee maker was getting its instructions from somewhere else. In the worst case, maybe 192.0.2 is a public network and some evil-doer could start flicking your lights on and off at random intervals, just to annoy you.

I see what you did there :)

The scenario you're painting is definitely a worthy concern! I would probably be freaking out of my lights were not mine to control anymore. And yeah, if i then figure out that limiting that subnet where bad actors could potentially have access too then i would slap myself in the face for not taking care of that right from the start.

However... None of this is a home network scenario. What you explained is a company network with potentially a public and private wifi hotspot. It multiple VLAN's for whatever reason. In those cases i understand the choice that has been made in Mosquitto 2.0.0. But for a home network these are hypothetical concerns that almost never are a real concern. For example, in a home network you often get a wifi access point from the same router that handles your wired points. There is no different subnet to detect as they all get their ip's from the very same DHCP server. So in a home network a user would be setting it as it were, only now the user needs to set it.

This does make me question.. For who is this change really? As it begins to smell like a corporate interest sneaking in here that, in the name of security, pushes this through. I don't say it's bad but i do question if corporate entities (the only place where this change makes sense in my opinion) are your biggest userbase. I'd be willing to bet that home IoT users are your biggest userbase (where this change makes no sense). And in those cases the 2.0.0 update just really broke their flow. For some users that might be a difficult endeavor to figure out. In my case the update came with just the normal distribution updates. In that very case nothing initially points to MQTT, it takes a lot of debugging to find that out.

All i'm saying is this. If this is a feature which is there to help the corporate environment then why make it default? Especially as MQTT on your loopback makes 0 sense to me for any environment (exceptions there). You can expect MQTT to be over the network (in fact, it's even described as a network protocol) so why the new default is to not do that is really not clear to me.

Please note I'm not speaking for Roger in my comments here, or really for anyone. I reserve the right to totally disagree with my own opinions. I also haven't gone back to look at the issue which was raised for this problem to see what was discussed there, although I do recall there being an issue some time ago.

I feel you're conflating my fairly mild advice about configuration file set-up and the change that caused your problem. I can pretty much guarantee that users with more complicated set-ups, whether they represent corporations are they're just little guys with a lot of smart toasters and different kinds of artisan bread, don't care at all about this one. They're not going to push for it, and they weren't affected by it when it came in.

Over the past decade or so, it's become very clear that there's way too much software with ports left open on the public internet. Wide-open trust is not an effective policy when faced with a world filled with scum-sucking weasels. A lot of this wide-openness has come from software which was designed to allow for easy initial deployment on the expectation that this deployment would happen behind a firewall or on a closed network, and last a short time until a favourable initial evaluation led to proper and secure configuration. That last part almost never happens, and the effect is vast arrays of bot nets, massive and successful ransomware attacks, and widespread loss of data. At some level, this problem puts the viability of the internet at risk.

So, there's a lot of pressure to make software be secure by default. Some of this probably comes from cloud providers, whose services are put at risk any time someone spins up a vm and starts an insecure application on it, but I suspect it's mostly from people who are genuinely concerned about the future of networking, and from poseurs trying to pad the "security expert" part of their CVs by opening CVEs based on well-known problem patterns. Just because they're poseurs does not mean they're wrong, though, and ultimately, it's important for software, and especially software that's aimed at people who don't want to spend a lot of time worrying about the security implications of their choices, to be as closed as possible by default.

Some consideration along that line led to this change. Ultimately, for this software to be secure by default, it has to do what it's doing now.

Thank you very much for your opinions and views on this! It's much appreciated.

I do get the "security paranoia" that would cause a change like this. I also definitely do get that me with my few lights is totally irrelevant in the bigger picture :)

But... I can't shake the feeling that, for the IoT purpose, this software is now effectively useless by default. That might sound harder then i mean it. People writing guides/howtos or even youtube videos on setting up IoT devices with MQTT will - eventually - find this change too and mention it in their content. In time, when searching for MQTT and IoT, it will probably be normal to add a tiny bit of configuration to make it work for their case.

I suppose the only useful case by default right now is when you have MQTT traffic internally within your own pc. And frankly, for internal pc communication MQTT makes absolutely no sense at all.

I'm just thinking out loud here for a bit, but it really makes me wonder what the default usecase is. I can't think of any.

It's just not reasonable to listen to * by default, no matter how convenient it is. That's like a new car being startable by anybody until you configure it. The default should perhaps be to refuse to run until a configuration choice is made.

However, this issue is not actually a bug report and I think it would be best if the submitter closed it as there are too many non-bug items open in the tracker. (Said as a fellow user; I don't have ticket gardening authority.)

I get your analogy but i don't think it's entirely fair.

To continue your car analogy, I'd describe it like this. It used to be a wide open car that anyone can enter. Now it's a closed down car that you can only enter if you have the key. I'm asking for a car sensor to automatically open if the sensor detects the user is authorized.

To map that on mosquitto. I'm asking it to be open within the LAN scope, not for the outside world. The mapping doesn't perfectly match the analogy but you get the point.

As for closing this issue. Yes, ill do that. I also get that from a publicity point of view the current state is "socially accepted" and that opening it by default to the localhost will be seen by some media as "security hole" who would gladly burn mosquitto for that. It's a delicate balance where the user always loses.

I can see your point somewhat. The basic issue is that the network used to be a nice place and now isn't -- I used to use a network-connected timesharing system that did not even have passwords! This has led to a changing of defaults towards security -- and yes that means with a negative usability impact -- over the years. I've seen it in lots of programs. The default change was very prominent in the release announcement at https://mosquitto.org/blog/2020/12/version-2-0-0-released/ but I get it that users of packaging systems might get an update without seeing that.

eclipse-mosquitto / mosquitto

I've just wasted 2 days because of 2.0.0 new security (IoT purpose here) #2302