MQTT/NodeRED down due to roof leak

[MatthewCroughan]

On the network we had:

• 10.0.100.1 MQTT
• 10.0.100.2 NodeRED

This has been brought down after a leak in the roof caused water to drip down onto the Raspberry PI's running these services. These servers can be re-provisioned using Balena via https://github.com/DoESLiverpool/balena-mosquitto https://github.com/DoESLiverpool/balena-nodered

I intend however to put these services on Doesnix instead so they are written as code and are more useful and reproducible, for me and others into the future.

• [x] MQTT is alive again!
• [x] NodeRED is alive again!

Site of Roof Leak

[johnmckerrell]

Is this a second leak or have they been down since the 9th July? What has actually died? PSU, Pi, SD card, all of the above?

[MatthewCroughan]

@johnmckerrell Nothing has died as far as I can tell, so no data lost.

I just want to take this opportunity to define it all as code, since currently it's not really defined as something that can be properly re-provisioned without reliance on a third party (balena), which is why I'm making this issue to document it.

[MatthewCroughan]

@johnmckerrell > Is this a second leak or have they been down since the 9th July?

It has been down since 9th July, I have just gotten around to making this issue as a form of documentation.

[amcewen]

Are you suggesting moving all the services onto the PC at your desk, or moving the Pis to run from NixOS?

Would it be possible for us to move mqtt.local to be the actual MQTT broker, rather than the Node-RED instance? It seems rather bizarre that I can't put mqtt.local into an MQTT client...

[MatthewCroughan]

@amcewen

Are you suggesting moving all the services onto the PC at your desk, or moving the Pis to run from NixOS?

I'm not necessarily suggesting everything run on one box, but for now that would be the implementation yes. Once it's in Nix code we can use that anywhere, on any piece of hardware, such as a Pi, though I just wanted to get it up and running today without faffing around with Pi's, bootloaders, and armv7, although if anyone has a Pi4 lying around I would be able to get that running quickly since that's arm64 which means it's properly supported.

This configuration I'm providing even provides a good bit of documentation if you wanted to use it outside of Nix, since it tells you how to run the service. For example, I've defined Mosquitto like this currently.

{
  networking.firewall.allowedTCPPorts = [ 1883 ];
  services.mosquitto = {
    enable = true;
    users = {};
    host = "0.0.0.0";
    allowAnonymous = true;
    aclExtraConf = ''
      topic readwrite #
    '';
  };
}

Which tells us that in order to run Mosquitto (2.0), you have to "allow anonymous". ensure the ACLs let you read/write every topic, ensure it's bound to 0.0.0.0, and that 1883 TCP is allowed, which is a form of documentation in telling anyone how to reuse the config even in a different system.

Would it be possible for us to move mqtt.local to be the actual MQTT broker, rather than the Node-RED instance? It seems rather bizarre that I can't put mqtt.local into an MQTT client...

That was just mis-configuration on my part which you can see here: https://github.com/DoESLiverpool/balena-nodered/blob/master/mdns-publisher/start.sh The nodered pi was always broadcasting mqtt.local, seemingly for the past X months! I've since unplugged that machine, since I only intended to grab its flows.json. If you ping mqtt.local now you should find that it is being broadcast from 10.0.100.1, which is running on doesnix inside of a container with its own IP thanks to the commits I just pushed. Namely https://github.com/DoESLiverpool/nixcfgs/commit/f0647c3be7d38f863ab27346fe72669da9775592

[MatthewCroughan]

I have moved the system physically to reside next to the TV near the entrance of DoES, can be seen in the center of this image in the laser cut case I made for it.

[amcewen]

The nodered pi was always broadcasting mqtt.local, seemingly for the past X months!

There was a reason I asked the question you know :wink:

more useful and reproducible, for me and others into the future

I agree with the "for me" (when the me is @MatthewCroughan :grin:) bit of that statement. Some pointers on how to get started with NixOS would be useful, for anyone who needs to do any maintenance on the system in the future.

[MatthewCroughan]

Some pointers on how to get started with NixOS would be useful

Sure, and this would be true of anything, at some point in time. (Docker, Ansible, etc were not well understood in the past). My goal here is to try to give some examples with the git commits I've been doing, which I am hoping are well written and good examples. I've put extra content in the messages of each commit message which describe exactly what is happening in each.

Though, I can't really think of a good way to suggest that people get started, other than to suggest they install the Nix package manager on their own distribution (Be it Ubuntu/Fedora/Arch or whatever), and start seeing what can be done.

I will think about how to best do this, but I am not sure about it right now, since obviously not everyone is as passionate about it as me, may not have the time, etc.

[Robotorium]

Okay sure, but then is that perhaps a point against using it?

Op ma 26 jul. 2021 om 19:19 schreef MatthewCroughan < @.***>:

Some pointers on how to get started with NixOS would be useful Sure, and this would be true of anything, at some point in time. My goal here is to try to give some examples with the git commits I've been doing, which I am hoping are well written and good examples. I've put extra content in the messages of each commit message which describe exactly what is happening in each.

Though, I can't really think of a good way to suggest that people get started, other than to suggest they install the Nix package manager on their own distribution (Be it Ubuntu/Fedora/Arch or whatever), and start seeing what can be done.

I will think about how to best do this, but I am not sure about it right now, since obviously not everyone is as passionate about it as me, may not have the time, etc.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-886924011, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXT43RJIRJ55RVISFT3TZWRMTANCNFSM5BACGMGA .

[MatthewCroughan]

@Robotorium

I'm not sure. I don't think so, because there is not necessarily anything better. Because this setup is reproducible whereas others typically are not.

Reproducible means that it can be brought back up again after X years, rather than someone making a machine one time with their fingers on a keyboard and that input being lost after the first time, which is something I'm passionate about resolving and want to work towards.

[MatthewCroughan]

Worth noting that nobody is forced to use anything, or rely on services I create on the network, because it's a network anybody can add their own services, this is just here for documentation, there's no officiality or "we should use this thing" here! I'm just documenting what I'm doing.

[MatthewCroughan]

In order to get NodeRED back up and running, I'll be adding node-red into the "services" of NixOS itself, upstream. See progress here.

https://github.com/NixOS/nixpkgs/pull/116578

[amcewen]

Worth noting that nobody is forced to use anything

That's not true though. Services that are shared and available for everyone to use should also be something that the community can continue to maintain even if the person who set it up isn't around. If that isn't going to be the case with the MQTT and NodeRED stuff you're setting up now then I don't think it should be the main MQTT broker in the space.

If you'd like an example of the sort of pointers I'm talking about, maybe have a look at my MQTT on a Pi ansible scripts.

Whoever will be trying to follow it will be wanting to solve a problem, like "the computer has died and we need to get things running again" or "it doesn't have feature X and it's hostile by design to any other way of adding that than doing something with something called nixos". They'll be trying to solve a problem, not learn nixos. Maybe they'll like nixos and want to use it more afterwards, but that will be afterwards, and might not happen.

[Robotorium]

Much as someone called nixinator clearly has a deeply unbiased opinion to share on NixOS -

The point of shared equipment is that people be able to use it. If it can't or won't be documented in such a way as to be useable by those people, then it's a dead end, because it stops being shared equipment.

Not going to respond to the rest because I haven't worked out an inoffensive way yet (and I never will).

Op di 27 jul. 2021 00:32 schreef nixinator @.***>:

I've seen that argument when i was deploying linux servers over windows 30 years ago. Linux was deemed 'not fit for production' even though it was more stable, had less downtime etc etc. Linux/Windows are in a evolutionary dead end, nixos solves a lot problems, but gives you some new ones. One of those problems is you got learning something new, experimenting, playing, having fun.... If you can't do that in a maker space, then where can you do it? So if people don't want to learn nixos, maybe they don't want to learn anything new. So the status quo remains. I guess it's a fact that if you always do what you always did, you'll always get what you always got. If your happy to use word soup deployment tools, the abomination that is docker, ansible , puppet , chef and the rest of that cloud native garbage, then you can... but they you will never realize they are garbage, but that's what your familar with, it's what the world uses...but what if the world is very very wrong, what if there's something better, but you have to learn something new. It will get you a job as the same companies running the world right now, but further than that ... ziltch.

The whole Debian ecosystem especially the networking on pi is complete insanity, making the same mistakes that have been going on for decades, do you really want young people making the same mistakes we made over and over again. If software is eating the world, lets make it software that actually coherent and solves problems.

Adding a 'cool' ansible script, and all that yaml rammle is like putting a band aid over severed limb.

you've got your tools, you've got your way of doing things, you're familiar with those abstraction, so would like everyone to learn them, I'm telling you those abstraction are not the way forward.

Software deployment , builds , maintainability are all going down the swanny, please correct me i'm wrong.

You need to install windows on your pi, and be done with it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887097423, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXT4BVGWYRELJ6XZP5DTZXWAVANCNFSM5BACGMGA .

[nixinator]

Windows it is then!!!!

i'm not a someone, i'm a AI robot from the future....

[Robotorium]

Cool, you enjoy that :)

Op di 27 jul. 2021 00:53 schreef nixinator @.***>:

Windows it is then!!!!

p.s. nix runs on windows too. :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887104415, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXQ7MIMYM7QI4F6XSUTTZXYOJANCNFSM5BACGMGA .

[nixinator]

Are you a big fan of ansible then? how do you configure your GNU/LINUX systems? I'm always open to new idea's....

[Robotorium]

Mate, you're really not having the conversation you think you're having.

Op di 27 jul. 2021 01:00 schreef nixinator @.***>:

Are you a big fan of ansible then? how do you configure your GNU/LINUX systems? I'm always open to new idea's....

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887106993, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXWD2MUK3JCGV2R6EW3TZXZIVANCNFSM5BACGMGA .

[nixinator]

ah, i take it your don't know anything about this stuff, sorry, i thought you were a developer. This mumbo jumbo is what we do ... i like your blog.

[MatthewCroughan]

@amcewen

This issue is supposed to be about documenting the return of the services, I didn't really want to compare/contrast anything in this issue, maybe we can discuss in person at some point, if you are interested or raise another issue about it, or use twitter, etc. Sorry if I've amped people up. (It's probably too late now, so I'll respond below).

In response to the Ansible setup you linked: The instructions state to install Raspbian lite of an unknown version, which will also end up giving us an unknown version of Mosquitto. It also requires network access to download things, where none of the stuff downloaded is compared against a hash (in your yaml file for example). So there's no guarantees that your Ansible script would result in the same output twice, and no way within the system of knowing if downloaded artifacts ever changed compared to what we expect.

In a nutshell, Nix solves the problems outlined above, by being reproducible and not relying on binaries distributed by third parties, only on source code. The flake.lock file locks our configs into a specific version of nixpkgs similar to how you might have package dependency lock systems for Rust/Golang. This means our systems will come out identical every single time we build them, by the hash of all inputs. That's one of a few reasons why it's worth using.

As an example, if you want to build a whole VM of doesnix to test, you can do this: nix build github:doesliverpool/nixcfgs#nixosConfigurations.doesnix.config.system.build.vm or of the Vinylcutter (it'll load Gnome/Inkscape with Inkcut and be exactly the same as the physical system if you boot it graphically) nix build github:doesliverpool/nixcfgs#nixosConfigurations.doesnix.config.system.build.vm

You can run those commands on anything, it's going to produce the same result on my laptop as it does for you on macOS, Ubuntu, etc. (A virtual machine image + qemu script to run it with will appear in ./result which is symlinked to /nix/store readonly).

You can build an ISO that you can flash to a disk, if that's preferred. It'll even compile everything from source if the default third party distributing cache (cache.nixos.org) were down, or we were making big changes to the software we want to use, like compiling Mosquitto with a different gcc flag or something. We don't even need the internet if we have a local cache of the source code on the network, which is something else that's trivial to set up.

I just created a repo that demonstrates how I can use Nix to build ISO images that can be flashed to an Orange Pi Zero here: https://github.com/MatthewCroughan/nix-opizero-systemimage-example

The TL;DR of that repo is that you can run nix build github:matthewcroughan/nix-opizero-systemimage-example#images.opizero and it'll build an SD image with everything we want inside of it, and indeed cross-compile any software needed for the system. The reason it needs to cross-compile is because nixos.org doesn't provide any cache for armv7l, and since I am using a custom version of Podman with a few patches.

I think this is probably one of the easiest ways to create embedded systems, especially in a Makerspace environment where requirements aren't as strict as they would be at my previous job in embedded.

It's possible I'm the only one at DoES that has this kind of enthusiasm for esoteric stuff like reproducible systems that work the same way every single time. I understand if you do not have the time to talk in person about anything, or generally don't share the enthusiasm and are not really that interested in what I'm saying, feel free to ignore me if you like :smile:

[nixinator]

consider yourself ignored.... :-)

[MatthewCroughan]

@amcewen PS: I understand that that's a verbose response to your comments, I also understand the message you're trying to give me which is to create some sort of instructions that guide from top to bottom, and I will do that soon(tm).

[Robotorium]

Matt:

The instructions state to install Raspbian lite of an unknown version, which will also end up giving us an unknown version of Mosquitto

Am I misreading, or does this imply that, if we need the config to upgrade to a newer version of $software, we have to manually edit all the configs (or rerun the command that creates them)? Is there an easy way to do that, or are we going to end up with ancient versions of Inkscape/Lightburn/whatever?

(Not always the worst in Inkscape's case but)

Op di 27 jul. 2021 02:48 schreef MatthewCroughan @.***>:

@amcewen https://github.com/amcewen PS: I understand that that's a verbose answer to your question, I also understand the message you're trying to give me which is to create some sort of instructions that guide from top to bottom, and I will do that soon(tm).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887145060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXW7QPEELXPUDVJNG2LTZYF5ZANCNFSM5BACGMGA .

[MatthewCroughan]

@Robotorium For that you simply run nix flake update on the repo, and it'll update the flake.lock file to refer to the new shiny up to date nixpkgs. You then git add flake.lock git commit -m "Updated the machine! Woo", and you can revert that flake.lock any time you want if, for example if you have discovered a bug that means the new version of Inkscape crashes. This is how new programming languages like Rust/Golang manage their dependencies. Nix is a programming language that produces Linux systems with software configured the exact way you want it to be, so it has a way of managing this that makes sense, just like those general purpose programming languages.

So we can travel through time on the system, and every single change made to the system is made through git commits like the ones you can see I've been making on this issue if you're on the Github website. If you're not, I'll paste an image here so you can see what it looked like.

There's nothing enforcing git commits, it's just good practice. It's not like you have to use git every single time you want to modify the system, just for testing. But you end up pushing the changes through git to show how you've changed the machine ultimately. Every point in the config can be rolled back and revisited.

[MatthewCroughan]

To further clarify @Robotorium, nixpkgs is basically the place where inkscape 2.0 currently is. Last week it was 1.9, the week before it was 1.8. So one typical way of managing this is just to cycle through those versions of nixpkgs rather than to say you want a specific version of inkscape, although you can do that as well if you ever wanted to. And yes, if you did that, it would be tedious to manage the version numbers just as you suggested.

[Robotorium]

I am in many places that I sometimes remember exist 😬

The main issue is it needs to be useable for everyone, and it needs to be documented to the point where it is that. Not that everyone has to be able to do it necessarily (I don't think everyone knows how to use Fred and I still haven't had a CNC induction because of the plague) but it has to be possible to look at our documentation and figure things out without having to learn NixOS from scratch (for example if someone were interested in learning some things but not others, or if we needed the lasercutter to work, etc).

Op di 27 jul. 2021 03:03 schreef MatthewCroughan @.***>:

@Robotorium https://github.com/Robotorium For that you simply run nix flake update on the repo, and it'll update the flake.lock file to refer to the new shiny up to date nixpkgs. You then git add flake.lock git commit -m "Updated the machine! Woo", and you can revert that flake.lock any time you want if, for example if you have discovered a bug that means the new version of Inkscape crashes. This is how new programming languages like Rust/Golang manage their dependencies. Nix is a programming language that produces Linux systems with software configured the exact way you want it to be, so it has a way of managing this that makes sense, just like those general purpose programming languages.

So we can travel through time on the system, and every single change made to the system is made through git commits like the ones you can see I've been making on this issue if you're on the Github website. If you're not, I'll paste an image here so you can see what it looked like.

[image: image] https://user-images.githubusercontent.com/26458780/127083044-94bdddc0-b303-4c68-b84b-e6a2b45f522b.png

There's nothing enforcing git commits, it's just good practice. It's not like you have to use git every single time you want to modify the system, just for testing. But you end up pushing the changes through git to show how you've changed the machine ultimately. Every point in the config can be rolled back and revisited.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887150270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXWIK6MUBNNVGGZBMZLTZYHXHANCNFSM5BACGMGA .

[MatthewCroughan]

@Robotorium TL;DR: the total process for upgrading a NixOS machine based on this configuration I've made would be:

1. nix flake lock --update-input nixpkgs
2. nixos-rebuild switch

That is all. You can then rollback that if you screwed up:

nixos-rebuild switch --rollback (Or reboot the machine and choose an older "generation" as it's called), every time you switch it puts an option in the bootloader that will allow you to rollback to before you screwed everything up.

After this, it's a choice whether you want to use git and push this change you've made to the flake.lock to github or not, where everyone see the changes you've made.

[MatthewCroughan]

The main issue is it needs to be useable for everyone, and it needs to be documented to the point where it is that.

I'll do my best. This is not going to be hard to maintain, I can promise this. Give me a few days and I'll see what I can do in terms of documentation of basic maintenance, such as upgrading the systems.

My idea would be that we simply have someone run nix flake update on this git repo and commit it. It is trivial to implement a system whereby every NixOS machine automatically updates to this git repo's definition of what the machine should be, every 5 minutes. It would be this line of code:

system.autoUpgrade.flake = github:doesliverpool/nixcfgs#vinylcutter

[amcewen]

@nixinator if you're the welcome committee for NixOS then the project is going to suffer as a result. You turning up every time nixos is mentioned to rant about Windows and how everything (you assume) that we do is wrong is the biggest argument to date for us not to use nixos.

[nixinator]

your not doing anything wrong, i'm frustrated that you find it hard that you will not even consider it , experiment and learn. This disappoints me and really goes against the ethos of tech spaces. Install it., give it a go, then write how bad it is.

This is not the first time you have said just use ansible, which is great and i use it myself.

You have problems with reproducable systems, people destroying configurations on that run your maker machines.

You know most maker spaces have that problem? Maybe does could help solve this problem for the world.

and boy is it nixos bad.

https://hands-on.cloud/why-you-should-never-ever-use-nixos/#why-you-may-want-to-use-nixos

avoid at all costs, and yeah i went a bit low to say 'install windows'.... I know does doesn't use windows and tried to avoid it, but if a machine is propertory , and runs windows software only, what can do you do? microsoft could always do we few extra $ in the bank after all.

keep doing what your doing which is pretty amazing..., and wish you well. leaks in the roof are never nice.

[Robotorium]

Nobody is not using it because it's "too hard." Bless your heart. People have other things to do, and shared equipment has to be workable/useable/documented so that people can do those other things.

Also, I don't think we're taking input on the ethos of makerspaces from someone who's just spent all of last night being offensively condescending about people who use windows. We want people who use windows to come in and make. There are plenty of windows users in DoES who have awesome skills, and even if they didn't have those skills we'd still want them to come in. They don't have to justify it, they don't have to excuse it, they can just use what they're happy using, and if they have trouble with a machine that isn't running something they're familiar with then there should be documentation, and there should be friendly, non-judgemental people who can help.

We definitely DON'T want people to feel like they have to be the l33test haxx0r or whatever bullshit, to be able to come in and be welcomed so they can do some making. That's not just against the ethos, it's poison to community building.

Op di 27 jul. 2021 11:42 schreef nixinator @.***>:

your not doing anything wrong, i'm frustrated that you find it hard that you will not even consider it , experiment and learn. This disappoints me and really goes against the ethos of tech spaces. Install it., give it a go, then write how bad it is.

You have problems with reproducable systems, people destroying configurations on that run your maker machines.

You know most maker spaces have that problem? Maybe does could help solve this problem for the world.

and boy is it bad.

https://hands-on.cloud/why-you-should-never-ever-use-nixos/#why-you-may-want-to-use-nixos

avoid at all costs, and yeah i went a bit low to say 'install windows'.... I'm sure you would never consider that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887406621, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXUTLJCPHZ6WZRWLOXDTZ2EP5ANCNFSM5BACGMGA .

[nixinator]

@Robotorium . Yep, we live in the world where most of tools are owned by pretty unethical corporations. If your uses are awawe of that and are willing to learn something new, that i see that a positive thing. I think promoting open source and free software in the maker community is the thing to do, but i agree it should not be force down peoples throats. If a machine needs windows to run and there is no alternative , then get that machine running.

The users should dictate what they want to run, something i probably forgot. So you completly right, no hacker bullshit. OI'm sure your have interesting opinions about free software and tools. I'm sure given the chance you would don't really want to contribute to one the most powerful corporations on earth. I think if your educating on that front, then the commuinity will decide whats best for them.

[Robotorium]

I haven't willingly run windows in decades and the only software I use that isn't open source is Lightburn.

It's nothing to do with me as an individual. It's about how to make people actually feel empowered to come in and engage with the space, and learn the things they want to learn, which is not by a) deriding the things they already know, or b) prescribing what they should learn and when.

Op di 27 jul. 2021 12:04 schreef nixinator @.***>:

@Robotorium https://github.com/Robotorium . Yep, we live in the world where most of tools are owned by pretty unethical corporations. If your uses are awawe of that and are willing to learn something new, that i see that a positive thing. I think promoting open source and free software in the maker community is the thing to do, but i agree it should not be force down peoples throats. If a machine needs windows to run and there is no alternative , then get that machine running.

The users should dictate what they want to run, something i probably forgot. So you completly right, no hacker bullshit. OI'm sure your have interesting opinions about free software and tools. I'm sure given the chance you would don't really want to contribute to one the most powerful corporations on earth. I think if your educating on that front, then the commuinity will decide whats best for them.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/1595#issuecomment-887419204, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHMPXQOGACJQUAPOQSCTGDTZ2HDLANCNFSM5BACGMGA .

[nixinator]

sounds positive. I've only seen the larger hack spaces , and helping out a one myself, so i'm still learning the ropes of what makerspaces do. Apart from all this code witchcraft, i do some other things to, in and around communities IRL. I tend to spend most of my time face to face with people, and not in github issues.

anyway i've got go now, i'm trying to get my 3d printer working again... hardware failure this time, not software!!!

I have found this discussion enlightening and it good to understand what users need from thier machines.

[MatthewCroughan]

The PR to NixOS for implementing the Node-RED service has went well and is probably going to be merged in a day or so.

It is now running on our network thanks to: https://github.com/DoESLiverpool/nixcfgs/commit/b8e574953b8d9b3cac95324aee2e2f932c9280b2 and https://github.com/DoESLiverpool/nixcfgs/commit/ee17627ddea5a31528c339a7472757c747580ac9

Everything on the network has re-established connection to the two services and seems to be working as normal.