Create a general-purpose hackable DoESLiverpool MQTT server

[goatchurchprime]

This is now a dominant protocol, so maybe it should be a component of our infrastructure, and to head off the Not Invented Here syndrome.

It should probably be on a spare RPi and documented on the wiki for anyone to access, or be on Doorbot if it is a robust enough system. It needs to have a fixed IPnumber.

[goatchurchprime]

See also #476

[goatchurchprime]

In particular, @ajlennon has a stack of these Sonoff ESP8266 switches, and there's a fine Arduino-based app that does all the basics at: https://bitbucket.org/xoseperez/espurna

Adrian has flashed the most recent version of espurna-1.10.1-itead-s20.bin onto a small S20 unit through a soldered serial port (with extra power from a 3V3 regulator).

Accessing using "picocom -b115200 /dev/ttyUSB9" allows me to see debug messages and find its IP number.

In the following example commands, the S20's IPnumber is 192.168.0.105 and my computer's IPnumber is 192.168.0.105

I've connected to its dashboard using: http://192.168.0.107/# user:admin password: bubbl1n0

This lets me set the MQTT broker IPaddress and port.

For debugability you can run your own MQTT broker like so: mosquitto -v -p 9005 and put in your own computer's ipnumber (192.168.0.105) into the Espurna dashboard webpage (port 9005) to see all the verbose messages as it connects and publishes notes when you press the button.

In another command window you see all the messages (basically button processes) from the device with: mosquitto_sub -h 192.168.0.5 -p 9005 -t "ESPURNA_A6F5FA/#" -v

If you want to control the device it's very useful to have spotted in the verbose messages that: Received SUBSCRIBE from esp8266a6f5fa 1513178879: ESPURNA_A6F5FA/relay/+/set (QoS 0) 1513178879: ESPURNA_A6F5FA/led/+/set (QoS 0) 1513178879: ESPURNA_A6F5FA/action/set (QoS 0)

Which means you can change the relay or the led with a command like so: mosquitto_pub -h 192.168.0.5 -p 9005 -t "ESPURNA_A6F5FA/led/0/set" -m "1" -d

---

Now (with the device connecting itself to the DoESLiverpool wifi) we are ready to set the MQTT broker (in the absence of our own one) to: test.mosquitto.org port: 1883

It is now possible to be running the following (listening) subscriber when it resets and reconnects: mosquitto_sub -h test.mosquitto.org -t "ESPURNA_A6F5FA/#" -v

and see that one of the messages is (very usefully): ESPURNA_A6F5FA/ip 192.168.0.107

This is going to help when you want to go onto its dashboard and change things or reflash the code.

(At times the S20 resets itself repeatedly (seen in the serial port) it eventually makes its own access point you get to through 192.168.4.1 with the above wifi password.)

When connecting to the default public MQTT broker you can change things using: mosquitto_pub -h test.mosquitto.org -t "ESPURNA_A6F5FA/relay/0/set" -m "0" -d mosquitto_pub -h test.mosquitto.org -t "ESPURNA_A6F5FA/led/0/set" -m "1" -d

There's some further workthroughs with: http://www.steves-internet-guide.com/mosquitto_pub-sub-clients/

[goatchurchprime]

There's a new sonoff POW unit that measures current*voltage power using an hlw8012 chip whose output interface seems to be a pulse-width frequency proportional to the power. If we get one this has been decoded in the espurna code: https://bitbucket.org/xoseperez/hlw8012

Meanwhile I am still battling with the STPM11 development board that performs the same purpose, but via a dodgy implementation of the SPI protocol.

[goatchurchprime]

@ajlennon did you have a stash of S20 plugs? the one I have is missing some covers so I can't install it without leaving some mains terminals bare.

[ajlennon]

Awesome. I dunno why I'm not getting notified when my @ajlennon pops up though. I'm happening across this stuff accidentally.

Yeah I've got some around. I'm keen to get the firmware upgraded to use this...

[ajlennon]

How do I work out which Sonoff units have this hlw8012 in?

[ajlennon]

Ah right. So the "POW" is the unit with it in. Looks good. Fair bit more expensive than the basic model.

[amcewen]

Given I had one of the old doorbot Raspberry Pi 1 boards to hand, I've set up a basic install of mosquitto on it (all scripted in Ansible, see MQTT-in-a-Box for details).

It will need securing - it accepts anonymous connections at the moment and doesn't run over TLS - but I figured I'd see how much it gets used first. It's available as mqtt.local on the default port of 1883.

[ajlennon]

How much data can I publish to it before I break it?

[amcewen]

I don't know. How much data do you want to publish to it?

[goatchurchprime]

Pinouts from the button end are Programmer, 3V3, Tx, Rx, Gnd, Nc

[goatchurchprime]

Instructions for commissioning:

Look up your hardware here: https://bitbucket.org/xoseperez/espurna/wiki/Hardware.md#markdown-header-iteadstudio-s20 Solder the pins on and connect to a serial port programmer (don't forget to hold the button in when you connect it up so it enters boot mode). Do "ls /dev/ttyU*" to find its address.

Look up your firmware here: https://bitbucket.org/xoseperez/espurna/downloads/

Flashing instructions are here: https://bitbucket.org/xoseperez/espurna/wiki/Backup.md https://bitbucket.org/xoseperez/espurna/wiki/Binaries.md esptool.py --port /dev/ttyUSB1 --baud 115200 write_flash -fm dout 0x00000 espurna-1.11.1-itead-sonoff-basic.bin The autoreboot doesn't work, so you have to powercycle it yourself.

Configuring is at: https://bitbucket.org/xoseperez/espurna/wiki/Configuration.md Connect to the ESPURNA wifi, wifi password is fibonacci, then go to 192.168.4.1 and login with "admin" and "fibonacci". Then it forces you to change the password, so change it to "bubbl1n0"

Add your network to the Wifi list on the webpage, then you can reconnect to http://espurna_a7a528.local/# to get to your device (don't forget it's protected by a login password)

[ajlennon]

Hi @amcewen. I can't seem to connect to http://mqtt.local:1883 from the internal network (or directly to 192.168.0.159:1883) although I can SSH to it.

Can I get my paws on an SSH authentication to have a go at adding Node-Red support please? :)

[ajlennon]

@goatchurchprime reading this

https://bitbucket.org/xoseperez/espurna/src

It looks like "Non-invasive current sensor using internal ADC or ADC121 or ADS1115"

That seems like we might be able to achieve the "drawing / not drawing" sensing we discussed?

[ajlennon]

@goatchurchprime when in SoftAP mode the ESPURNA_XXXXXX AP seems to need a password. Any thoughts?

[Edit: Ah that's 'fibonacci' too. OK]

[ajlennon]

And also I knew I'd seen those switches I mentioned somewhere.They are here too

"IteadStudio Sonoff Touch"

https://bitbucket.org/xoseperez/espurna/src

Details here: https://www.itead.cc/sonoff-touch.html

What about these in Neo-DoES?

[amcewen]

The server now has its own repo in the DoES github account - https://github.com/DoESLiverpool/DoES-IoT-server and that's just had an additional role added to install and run a NodeRED server. Ideally we'd move the generic Mosquitto stuff into its own Ansible role too, but I've not done that yet. Similarly I haven't added any login details on the NodeRED instance, @ajlennon, do you fancy adding that?

@ajlennon should you be able to connect to http://mqtt.local:1883? Surely you're mixing protocols there? http://mqtt.local:1880 will get you the NodeRED instance (now).

[ajlennon]

OK so NodeRed is up and running on @amcewen 's MQtt-in-a-box (was there by default!)

I've got a noddy flow running to debug output from the ESPURNA I have here.

@goatchurchprime we might want to define a topic space so we can have a root that is appropriate for DOeS devices - a well designed topic space makes all the diffference for Pub/Sub - e.g. if I want to filter on all SonOffs, all SonOffs of a particular type etc. etc.

[ajlennon]

As with smart watches people seem incapable of doing decent product design work for digital home switches.

Nothing really leaps out here

https://ideaing.com/ideas/best-smart-light-switches-plugs

There is an interesting comment about an IFTTT recipe to control heating based on weather though @goatchurchprime as you were discussing.

@amcewen made a good point earlier that it would be nice to have a dimmer switch kind of arrangement for the physical input. Which could then publish via MQtt or LoRA or whatever I guess

[ajlennon]

@ajlennon should you be able to connect to http://mqtt.local:1883? Surely you're mixing protocols there? http://mqtt.local:1880 will get you the NodeRED instance (now).

You're absolutely right - that was the problem!

[ajlennon]

So I got things going quite nicely. I have a node-red flow (on my server as it has to be public facing for Amazon)

I have a new "DoES Liverpool" skill to play with, which calls the NodeRed flow. Currently that's just working out what parameters were sent for the "intention" - i.e. what I told it - and is returning to text to be spoken back to me. Obviously we can next start to wire in some actions... I'm thinking current consumption and costs and stuff would be good. And when the coffee is ready of course.

So I used Nathan Chantrell's blog post to get through it which was really useful -

https://nathan.chantrell.net/20160328/amazon-echo-alexa-skills-kit-integration-with-node-red/

I had trouble with certs and put some notes in a comment which is presumably still undergoing moderation. Basically AWS doesn't seem to like the LetsEncrypt CA root.

[ajlennon]

Similarly I haven't added any login details on the NodeRED instance, @ajlennon, do you fancy adding that?

I think we may want to have a chat about if/how we would connect a public facing https:// endpoint to AWS to give skills access first?

[ajlennon]

The POW has arrived so will bring it in so we can try to get it flashed with Espurna and wire it up to an extension cable so we can try some current measurement @goatchurchprime.

[goatchurchprime]

@ajlennon It is flashed and operating at http://espurna_3b6f59.local/#

The Xmas tree is using 29W.

And it's using the mqtt.local (although the IP number had to be hardcoded into the config to work). mosquitto_sub -h mqtt.local -t "ESPURNA_3B6F59/#" -v

I've made my proposal for the device to interpret the measurements into events so we can derive complex behavior without any unreliable processing: https://bitbucket.org/xoseperez/espurna/issues/369/report-energy-use-per-event-on-the-emon

I'm wondering if one could also add more sophisticated mqtt subscribers directly into the espurna (like if-this-then-that) so that they automatically chain. Goodnight lamp without any node-red stuff.

One could chain a whole series of electric kettles so that when one finishes boiling the next one starts boiling, so there is always one forever on the boil.

[ajlennon]

Got one of these on order for adding in Alexa voice control

https://t.co/5HoLkMgR55

[ajlennon]

@johnmckerrell - @goatchurchprime and I would like to experiment a bit with Amazon Echo voice control.

I have Amazon "skills" connected to Node-Red flows which enables me to be pretty flexible in what can be done.

However for this to work there has to be a public-facing https:// endpoint for Amazon to connect to.

We do have an internal mqtt.local box on the DoES network which @amcewen kindly set up.

I think it would be very useful to be able to work with such a box on the internal network from a control and monitoring standpoint and to connect to that from Amazon Alexa services

I am wondering if it is possible (sensible/secure) to allow a proxied https:// call from Amazon servers through to the internal box?

Or if there's another option?

Or it if is a no go... :)

[goatchurchprime]

There are APIs for converting speech to text that you could put into the workflow once you can have isolated the audible command into an MP3 snippet:
https://cloud.google.com/speech/
We should try using that.

Implement this into a giant chunky Microphone-Of-Command which is the size of a football, or as a hands-free headset that you put on in order to conduct and insert comments into, say, the weekly laser cutter servicing checklist.

I would like to have a robot to order me around, telling me exactly what to do and making sure I am doing it right.

I am convinced the MOC is the fundamental solution of the control panel problem (where adding remote operation capabilities to devices ends up more than doubling the number of switches and indicators).

When you have voice control, adding new devices into the controlled system does not make things more complex. For example, coffeebot used to have a dial indicator of the amount of coffee remaining, and could have had a clock timer for when the coffee was last made. But with voice control, you'd say into the one MOC "How old is the coffee?" or "Read coffee status", and all the information would come back as a string of spoken descriptive words in English, instantly understood by the brain without any dial-reading interpretation.

Whenever we felt like it, we'd insert a new regexped command phrase to generate a chain of actions and feedbacks.

[Thingomy]

Please tell me we aren't considering having always on microphones hooked up to some shower of corporate overlords? I'm not into that and would have a strong desire to either defenestrate it, or tell it to order 2 tonnes of creamed corn on a regular basis.

No problem with others using or even developing such things, but I really dislike the idea of having my voice recorded routinely in a connected fassion and used for god knows what.

JR

On 15 Jan 2018 14:00, "Julian Todd" notifications@github.com wrote:

There are APIs for converting speech to text that you could put into the workflow once you can have isolated the audible command into an MP3 snippet: https://cloud.google.com/speech/ We should try using that.

Implement this into a giant chunky Microphone-Of-Command which is the size of a football, or as a hands-free headset that you put on in order to conduct and insert comments into, say, the weekly laser cutter servicing checklist.

I would like to have a robot to order me around, telling me exactly what to do and making sure I am doing it right.

I am convinced the MOC is the fundamental solution of the control panel problem (where adding remote operation capabilities to devices ends up more than doubling the number of switches and indicators).

When you have voice control, adding new devices into the controlled system does not make things more complex. For example, coffeebot used to have a dial indicator of the amount of coffee remaining, and could have had a clock timer for when the coffee was last made. But with voice control, you'd say into the one MOC "How old is the coffee?" or "Read coffee status", and all the information would come back as a string of spoken descriptive words in English, instantly understood by the brain without any dial-reading interpretation.

Whenever we felt like it, we'd insert a new regexped command phrase to generate a chain of actions and feedbacks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/616#issuecomment-357690450, or mute the thread https://github.com/notifications/unsubscribe-auth/AIulCzwxMs9k-LHqnHtAZP2JWPhpg-pJks5tK1nkgaJpZM4RAzOy .

[ajlennon]

I can't work out all these people who don't trust a piece of software in a box (quite possibly opensource, transparent software designed in house) that decodes audio on command but are happy to be surrounded by other peoples' laptops willy nilly

[goatchurchprime]

If you could find a speech to tweet app, it could be wired up through the cheerlights system.

@DefProc pointed to: http://jasperproject.github.io/ as something to install and make work.

[ajlennon]

So I best have a go at recapping for my own recollection if nothing else.

I made the mistake of trying to install Jasper from scratch which is my general preferred way to understand what's going on with code frameworks. Needless to say that proved to be a problem. I got Raspbian and a multitude of dependencies from the above link on there ok, including Jasper.

However the speech to text engines defeated me. On some general principles of privacy and internet independence I went with the only available offline option that doesn't need tuning. This proved complicated as the compilation of open-xxx kept locking up the RPi, I think due to lack of memory but even so it's a pretty painful failure mode. (instructions say compile as root, tried compiling as user, tried adding swap space. no joy)

After much ado I then stepped back to the prepared "this will work" image download and this promptly failed to boot on the v3 (nothing - no bootloader. I assume there's a v2 image and it doesn't work on v3?)

I moved onto some work by Matt Curry here

http://www.mattcurry.com/projects-2/the-jasper-project/jasper-project-useful-links/

Downloads were crawling along so I have downloaded at home and can report it is at least running and on the WiFi.

I now need to work out whether his "All In One" image supports the offline STT and TTS we want and find a way to do a simple test....

[goatchurchprime]

I've got a demo working in the office from the cardboard box Rpi where "oscilloscope on" will turn the Sonoff Pow on, which is now connected to the fan.

It's a set of function calls in the src/somthing.py file (in the editor) that enable these commands.

Obviously new commands ought to be programmable in... also through a voice interface.

Next task is to put the sonoff pow in line with the coffee machine and find out how to record the last spike of energy (probably on the sonoff itself if it was programmable, because it's a pain doing it in some nodered thing)... so that it can answer "How old is the coffee?"

Jackie is looking at what handsfree headset is going to work on the PiZero (which the google thing should still work with), which gives some degree of microphone privacy, because it's only talking when you are wearing the device.

[goatchurchprime]

@johnmckerrell Where's the issue/wiki page about the static IP address technology your friend was doing on a Rpi in the cabinet last night?

It was an idea to put the MQTT server (as mqtt.local) onto that same Rpi to save on unnecessary Rpi deployment. .

[skos-ninja]

Hi Julian, I will be in on Thursday evening if we want to set it up then

[goatchurchprime]

@skos-ninja Did you get mqtt.local in at all? (I tried pinging just in case, and nothing came up)

[goatchurchprime]

@skos-ninja have you left any instructions of how to log on to that loose Rpi so I can do it?

[johnmckerrell]

I have login details for the box. What is required? Just apt-get install mqtt ? As a crucial piece of infrastructure generally I’d like to limit the number of people who have access to the box.

On 14 May 2018, at 16:57, Julian Todd notifications@github.com wrote:

@skos-ninja https://github.com/skos-ninja have you left any instructions of how to log on to that loose Rpi so I can do it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/616#issuecomment-388868843, or mute the thread https://github.com/notifications/unsubscribe-auth/AABqawkV5brtxpbFZyBQJa2gKEGOPYIGks5tyalWgaJpZM4RAzOy.

[johnmckerrell]

I installed mosquitto and changed the hostname of the pi to mqtt.local which I've confirmed works remotely. Let me know if you can access it ok (and @skos-ninja would be good to confirm that the hostname change will be fine but it seemed ok).

[skos-ninja]

Yep this should be fine. I would have recommended getting the pi another IP and set it up on that though

[goatchurchprime]

I'm summarizing the state of play at: https://github.com/DoESLiverpool/wiki/wiki/MQTT-services

[skos-ninja]

To note here this is now not working again after it killed the SD card on the UniFi Pi. As such I would not recommend putting anything else on that Pi unless it is to do directly with networking.

This has now stopped progress on #598 as we can not setup the new switch properly as it can not be adopted by the UniFi software due to the SD card being dead.

I would recommend to @johnmckerrell that we find the old raspberry pi that was set up for this and install that again and not to reinstall it on the networking Pi.

[ajlennon]

Hey @skos-ninja - have we got a guest network connection that we can safely attach devices to for external visibility?

[skos-ninja]

Hi @ajlennon,

We do not currently have a way for you to access devices outside of the network right now.

We will probably not be doing this for a while as we still need to go through the steps of making this a secure but easy process.

[ajlennon]

OK - I was accessing my Linux build box back in the Hanover Street days and could really do with having this working again asap...

[johnmckerrell]

Do you have an external server you can access? You could use SSH port forwarding to enable this as an interim measure. So from your build server you would do: ssh myserver.example.com -L 22:localhost:2225

Then you can ssh into myserver.example.com, and from there SSH into your build box (something along those lines, can’t remember the order of arguments there).

On 1 Jun 2018, at 15:35, Alex Lennon notifications@github.com wrote:

OK - I was accessing my Linux build box back in the Hanover Street days and could really do with having this working again asap...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DoESLiverpool/somebody-should/issues/616#issuecomment-393899951, or mute the thread https://github.com/notifications/unsubscribe-auth/AABqa2A3mdeyY4vcIeB1LD_we8trR0-Bks5t4VEvgaJpZM4RAzOy.

[ajlennon]

I think I've been using does.mckerrell.net or something but I don't have the laptop to hand atm ?

Was recommended and thinking of using ngrok moving forward but wanted to make sure that wasn't going to cause a security issue

[ajlennon]

Ideally I'd like to be able to access my box again, but to have it on a DMZ network segment such that if (god forbid) anything went wrong any attacker would not have access to anything in a secure segment?

[skos-ninja]

This is exactly what I was just talking to @johnmckerrell about!

I'm looking to set this up this weekend so that we can start to port forward and it not be a security issue.

If you are looking to help I will be around tomorrow (2nd June) working on this.

[ajlennon]

Excellent. Great to hear we're thinking along similar lines! I've been away diving but am around and about for parts of the week

[amcewen]

I've recreated the setup documented in https://github.com/DoESLiverpool/somebody-should/issues/616#issuecomment-354555973 onto a micro SD card provided by @goatchurchprime and he's stuck it into a Pi (the one by the TV and the Liver Bird), so there's now an MQTT server on mqtt.local again.