Help needed to implement your code

happytm commented 4 years ago

I tried your code and I like the simplicity to display lot of data with very little effort. Now I do not use
Homie convention for my project but I will describe my project. Basically I have local standalone IOT network in star topology. There are many battery powered slave devices connected to an ESP8266 gateway which is mains powered and always on receiving sensor data from all slave devices. The gateway publishes these data via MQTT in following format:

received topic 'DeviceStatus/{"Location":"Livingroom","SingnalStrength":-82,"DeviceMode":0,"DeviceIP":26,"WiFiChannel":7,"SleepTime":1,"UpTime":138}

received topic 'SensorValues/{"Location":"Livingroom","Volts":2.62,"Sensor1":"Temperature","SensorValue1":1,"Sensor2":"Humidity","SensorValue2":2,"Sensor3":"Pressure","SensorValue3":124,"Sensor4":"Light","SensorValue4":67}

received topic 'DeviceStatus/{"Location":"Bedroom1","SingnalStrength":-54,"DeviceMode":0,"DeviceIP":16,"WiFiChannel":7,"SleepTime":1,"UpTime":139}

received topic 'SensorValues/{"Location":"Kitchen","Volts":2.62,"Sensor1":"Temperature","SensorValue1":1,"Sensor2":"Humidity","SensorValue2":2,"Sensor3":"Pressure","SensorValue3":129,"Sensor4":"Light","SensorValue4":66}

There is also control part in my code which allow me to control all these slave devices by issuing MQTT command using following format: Typical MQTT command is like: command/36/06/16/26/36/46/ where "command/" is topic name and "36/06/16/26/36/46/" is payload.


        Command1 = Device ID Number -       device ID must be 2 digits end with 2,6,A or E. See https://serverfault.com/questions/40712/what-range-of-mac-addresses-can-i-safely-use-for-my-virtual-machines.
                                            use any of following for devie ID ending with 6.
                                            06,16,26,36,46,56,66,76,86,96,106,116,126,136,146,156,166,176,186,196,206,216,226,236,246 etc.
        Command2 = Command type  -     value 1 to 9 is reserved for following commands(must have 0 as first digit):

                                       01 = digitalWright or analogWrite.
                                            Example command payload 36/01/00 0r 01/ for digitalWrite.
                                            Example command payload 36/01/02 to 256/ for analogWrite.
                                       02 = digitalRead.
                                            Example command payload 36/02/01 to 05 or 12 to 16/ 
                                       03 = analogRead, 
                                       04=  Reserved, 
                                       05 = Neopixel etc.
                                            Example command payload 36/05/01 to 05 or 12 to 16/00 to 256/00 to 256/00 to 256/
                                       06 = change sensoor types.First byte must be target device id and 
                                            second byte must be 06 (sensor type voltage). Rest of 4 bytes (each ending with 6) can be changed according to hardware setup.
                                            Example command payload 36/06/16/26/36/46/.

                                       07 = change apChannel, 
                                       08 = change sleeptime. 
                                            Example command payload 36/08/00 to 255/ (Sleep Time in minutes).  
                                       09 = Activate alternative code for OTA,Wifimanager ETC.
                                            Example command payload 36/09/00 or 01/(01 to activate alternative Code).

                                            value 10 to 20 is reserved for following commands:
                                       10 = change define DUPLEX, 11 = change define SEURITY, 12 = change define OTA, 13 = change define uMQTTBROKER etc.

        Command3 = Command  pinNumber  -    pinNumber in case of command type 01 to 04 above. Neopixel LED number in case of command type 05.
                                            Predefined number to represent value of command type 11 to 20 above.
                                            00 or 01 to represent false/or true for defines in case of command type 21 to 30. 

        Command4 = Command pinValue  -      00 or 255 in case of command type 01 (digitalWrite & analogWrite)  or RED neopixel value in case of command type 05.

        Command5 = Command extraValue1  -   00 to 255 for GREEN neopixel in case of command type 05                        
                                            or sensorType value in case of command 06.
        Command6 = Command extraValue1  -   00 to 255 for BLUE neopixel in case of command type 05 
                                            or sensorType value in case of command 06. 

> I wanted to use these data and display in Node-Red dashboard adding date and time to it. Can you please help with setting it up in Node-red.

My code is at https://github.com/happytm/BatteryNode .

Thanks.

happytm commented 4 years ago

Also noticed that Node-Red complains (Error: input not a json array) with your example design file .

Thanks.

Christian-Me commented 4 years ago

Just find your issue here. As I'm in home office I so far behind in checking all my inboxes. Just busy today but perhaps take a look here (especially the translate function node): This part of the flow uses incomming json data and converts them for the table. You can find a complete list of properties in the readme.md or on the info panel of the flow: Just feed in your data as JSON and sort the relevant properties into the reserved fields. (If you do not want to modify the table in the beginning)

The control part is certainly possible (perhaps with context menus) but not implemented here. I use ui-table as in and output in my sprinkler system (work in progress):

happytm commented 4 years ago

Thank you for help. It will be fun making my table.

Christian-Me commented 4 years ago

Don’t know how good your js skills are - sorry most if my flows are very js based. Ask if you need any help. Should not be an bog task to achieve your goal. ESP now is a interesting approach but unfortunately no mesh capability. Will do all my future battery powered devices on zigbee as nowadays good of the shelf and some interesting boards are available.

happytm commented 4 years ago

I am sure I will ask questions as I am not very good at javascript. I think you misunderstood my code. I do not use ESPNow but I use probe request as my custom protocol. It is better than ESPNow for power consumption. It could be even used as a MQTT repeater to extend the range of whole network. It uses only around 50 ms to send data from slave to master as compared to 5 to 10 seconds for WiFi. No need to manually charge small 100 mAh LIR2450 coin cell battery for life of battery or solar cell if LIR2450 coin cell battery and 53mmx30mm small solar cell linked in my repo hardware section.

Please try following 2 sketches if you have 2 ESP8266 devices:

1) https://github.com/happytm/BatteryNode/blob/master/ProbeReceiver.ino (Master or Gateway)

Here you will be able to see how messages are received from slave devices via probe request and 
converted to json format. This code include MQTT broker so you do not need to use any cloud 
broker. On receiving data from any slave device the MQTT broker publishes this data 
immediately.

2) https://github.com/happytm/BatteryNode/blob/master/ProbeSender.ino (Slave)

Thanks.

happytm commented 4 years ago

I am using your flow with minor modifications to table and feeding it from MQTT in node.

My flow file is here : https://github.com/happytm/BatteryNode/blob/master/test/ui-table.json

I have created 6 inject nodes to simulate published data from 3 of my ESP8266 devices.

I am using ESP8266 code from here for publishing MQTT messages : https://github.com/happytm/BatteryNode/blob/master/ProbeReceiver.ino

Now I have problem with above flow. My table update sometimes but not every time it receive MQTT message. Also Can you look at my function node and see if it is correct or I need to do something more to implement it.

Thanks.

Christian-Me commented 4 years ago

Sorry I (again) didn’t received your comments - somehow the new GitHub app isn’t reliable. Will look into it the next days.

happytm commented 4 years ago

Thank you for your response.

I just uploaded my current flow.json file if you want to look at it.

https://github.com/happytm/BatteryNode/blob/master/test/flow.json

I think I was able to resolve some of the issues but I still want to send commands via MQTT to my gateway device from the dashboard.

Thanks.

Christian-Me commented 4 years ago

Hi. I quickly looked into the flow. Beside I do not use sql databases or the mosca broker I thing I need some help to identify your issue.

How / where do you want to trigger the commands? because sending a msg like msg={topic:"command",payload:"36/06/16/26/36/46/"}; to the broker is proberbly not your issue.

BTW: be careful with any extra / for the topic. MQTT is sometimes a little bit strange. an / in the beginning or end is valid but produces a topic level with an empty name because every / produces a new level

Typical MQTT command is like: command/36/06/16/26/36/46/ where "command/" is topic name and "36/06/16/26/36/46/" is payload.

happytm commented 4 years ago

I tried to make mock up of how control part should behave. I put delay between selecting from drop boxes and actually sending MQTT message to allow for time to add some extra command to payload if we want to send longer custom payload.

I am sure there are better ways to do it. Following is my flow:

Christian-Me commented 4 years ago

Hi,

I looked into your flow.

There is also control part in my code which allow me to control all these slave devices by issuing MQTT command using following format: Typical MQTT command is like: command/36/06/16/26/36/46/ where "command/" is topic name and "36/06/16/26/36/46/" is payload.

I still have my problems with your command structure. MQTT let you do anything you want with topic payload combination but to speak in AGC (Apollo Guidance Computer) terms topic is the verb and payload is the noun. I would see the topic as command1 to command3 and payload as command4 to command5 as parameters. The problem i see is that the client (i.e. Node-RED) needs to know a lot of internals of your device like the gpio pin number. If you change this on your "endpoint" you have to change all your implementations.

Perhaps think about to change this to a more abstract level like

msg={
  topic: "myBridge/myDevice/led1/hue",
  payload: "300,50,50"
}

So your bridge can subscribe to mybridge/# and dispatch the messages to your devices. The device should then know where led1 is connected, do a conversion from hue to RGB or RGBW and send it to your neopixel.

The only disadvantage I see is that you have to roll out individual firmware to your devices. If you don`t like this perhaps the bridge can transform the commands.

I`m personally a big fan of the homie convention which fits all my needs.

Back to your problem.

I have no experience with the join node. I would prefer a function node collecting all parameters and then a send button to actually post your message. Done a quick function node ... perhaps this could help

[{"id":"1f46c197.8a7e9e","type":"debug","z":"cc1a1f30.80cc5","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":951,"y":442,"wires":[]},{"id":"4fa8a11d.8577c","type":"ui_dropdown","z":"cc1a1f30.80cc5","name":"Select Room","label":"","tooltip":"Select Room ","place":"Room","group":"dbc163d5.e6b2e","order":1,"width":3,"height":1,"passthru":true,"options":[{"label":"Livingroom","value":"06","type":"str"},{"label":"Kitchen","value":"16","type":"str"},{"label":"Bedroom1","value":"26","type":"str"}],"payload":"","topic":"room","x":359,"y":355,"wires":[["b20f3b7f.994b28"]]},{"id":"d2542af7.e9a418","type":"ui_dropdown","z":"cc1a1f30.80cc5","name":"Select GPIO/Sensor Type","label":"","tooltip":"Select Pin Number or Sensor Type","place":"GPIO No.","group":"dbc163d5.e6b2e","order":3,"width":3,"height":1,"passthru":true,"options":[{"label":"01","value":"01","type":"str"},{"label":"02","value":"02","type":"str"},{"label":"03","value":"03","type":"str"},{"label":"04","value":"04","type":"str"},{"label":"05","value":"05","type":"str"},{"label":"06","value":"06","type":"str"},{"label":"12","value":"12","type":"str"},{"label":"13","value":"13","type":"str"},{"label":"14","value":"14","type":"str"},{"label":"15","value":"15","type":"str"},{"label":"16","value":"16","type":"str"},{"label":"26","value":"26","type":"str"},{"label":"36","value":"36","type":"str"},{"label":"46","value":"46","type":"str"},{"label":"56","value":"56","type":"str"},{"label":"66","value":"66","type":"str"},{"label":"76","value":"76","type":"str"},{"label":"86","value":"86","type":"str"},{"label":"96","value":"96","type":"str"}],"payload":"","topic":"gpio","x":399,"y":435,"wires":[["b20f3b7f.994b28"]]},{"id":"9a4b640f.d4f6e8","type":"ui_dropdown","z":"cc1a1f30.80cc5","name":"Command Type","label":"","tooltip":"Select Command Type","place":"Command","group":"dbc163d5.e6b2e","order":2,"width":3,"height":1,"passthru":true,"options":[{"label":"Set Sensor Type","value":"06","type":"str"},{"label":"Toggle GPIO","value":"01","type":"str"},{"label":"Set Level","value":"02","type":"str"},{"label":"Digital Read","value":"03","type":"str"},{"label":"Analog Read","value":"04","type":"str"},{"label":"Neopixel","value":"05","type":"str"},{"label":"Set Wifi Channel","value":"07","type":"str"},{"label":"Set Sleep Time","value":"08","type":"str"},{"label":"Set Device Mode","value":"09","type":"str"}],"payload":"","topic":"type","x":369,"y":395,"wires":[["b20f3b7f.994b28"]]},{"id":"5b1f855a.cd73cc","type":"mqtt out","z":"cc1a1f30.80cc5","name":"","topic":"command","qos":"","retain":"","broker":"e20b5deb.be4a4","x":961,"y":493,"wires":[]},{"id":"6519f306.afebdc","type":"ui_button","z":"cc1a1f30.80cc5","name":"Reset","group":"dbc163d5.e6b2e","order":4,"width":2,"height":1,"passthru":false,"label":"Reset","tooltip":"Reset the command","color":"","bgcolor":"","icon":"","payload":"reset","payloadType":"str","topic":"command","x":339,"y":646,"wires":[["b20f3b7f.994b28"]]},{"id":"622a3922.17ebf8","type":"ui_button","z":"cc1a1f30.80cc5","name":"Send","group":"dbc163d5.e6b2e","order":6,"width":2,"height":1,"passthru":false,"label":"Send","tooltip":"Reset the command","color":"","bgcolor":"","icon":"","payload":"send","payloadType":"str","topic":"command","x":339,"y":595,"wires":[["b20f3b7f.994b28"]]},{"id":"b20f3b7f.994b28","type":"function","z":"cc1a1f30.80cc5","name":"","func":"var data = context.get('data') || {\n    room:\"06\",\n    type:\"06\",\n    gpio:\"01\",\n    command:\"\"\n};\nif (msg.topic && msg.topic === \"command\") {\n    switch (msg.payload) {\n        case 'reset':\n            data={};\n            break;\n        case 'send':\n            msg.topic='command';\n            \n            msg.payload='';\n            msg.payload+=data.room+\"/\";\n            msg.payload+=data.type+\"/\";\n            msg.payload+=data.gpio+\"/\";\n            msg.payload+=data.parameter;\n            \n            return msg;\n    }\n} else {\n    data[msg.topic]=msg.payload;\n}\ncontext.set('data',data);\n","outputs":1,"noerr":0,"x":696,"y":459,"wires":[["1f46c197.8a7e9e","5b1f855a.cd73cc"]]},{"id":"4305f326.fe2c8c","type":"ui_text_input","z":"cc1a1f30.80cc5","name":"","label":"Enter your Command","tooltip":"Enter your command Payload here","group":"dbc163d5.e6b2e","order":5,"width":9,"height":1,"passthru":true,"mode":"text","delay":"0","topic":"parameter","x":389,"y":493,"wires":[["b20f3b7f.994b28"]]},{"id":"dbc163d5.e6b2e","type":"ui_group","z":"","name":"Table","tab":"91766151.e711","disp":true,"width":11,"collapse":false},{"id":"e20b5deb.be4a4","type":"mqtt-broker","z":"","name":"Localhost","broker":"Localhost","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"91766151.e711","type":"ui_tab","z":"","name":"newTable","icon":"dashboard","disabled":false,"hidden":false}]

happytm commented 4 years ago

Thank you for your help. In the past I looked in to homie convention and spent quite some time understanding it but somewhere along the line I got confused with it and gave up. It is time to re read it again and try to understand it with fresh mind. I started with join node because it is the easy way to implement control part without knowing javascript. I think I will be able to follow javascript code you provide with function node and try to implement control part with MQTT.

I am becoming fan of tabulator as I read more and more about it. I also looked in to Pivottable example linked at https://pivottable.js.org/examples/montreal_2014.html . I think eventually I want to implement one of this framework directly on my ESP32 gateway device without using node-red (Raspberry PI Zero W) at all for truly local standalone network monitor and control.

I am working on control part of code for standalone implementation using jquery and tabulator which will provide same look and feel like node red implementation. Attached here is work in progress for that code.

Dropdownhtml.txt optionsjson.txt

Thanks.

Christian-Me commented 4 years ago

I would definitely take a look into the homie convention. I wrote the homie plugins for ESPEasy and the contrib nodes for Node-RED. There is a good C++ ESP library available too I used for my Christmas Tree 6h challenge (posted in the form). I even use homie/mqtt broker as my main state storage for all of my devices and services. The broker is the center of my installation - There is no other central control role. A light switch for example sends the command to the dimmer module directly (via the Broker .../set topic) Node-Red and the dashboard (running on multiple instances) are only informed about the state change. I offload basic functionalities like timers, or thermostats to the device and let all participants only set the values like setTemperature. And with mqtt explorer you have a good live view on the state of all your devices and services. And no obscure codes / structures: all human readable in example: lights/dimmer1/neoPixel1/hsv

happytm commented 4 years ago

Can you point me to one simple .ino file using homie which reads one sensor like BME280 and send MQTT messages for sensor data. It will be easier for me to follow 1 simple example and follow it to build more sensors ontop of it. Sometime I get overwhelmed reading too much of documentation.

Christian-Me commented 4 years ago

Hmmm ... you can use the dev branch from ESPEasy including the Homie controller and the Homie device plugin.

Here I've done an example for a sonoff S20 socket using ESPEasy:

https://github.com/Christian-Me/node-red-contrib-home/tree/master/ESPEasy/Sonoff-S20-EU-V0.0.0 use the latest firmware (4MB) form here https://github.com/Christian-Me/node-red-contrib-home/tree/master/ESPEasy/firmware

or my christmas tree project with a homebrew firmware (vscode + platform.io). Sorry I don't use arduino ide any more ... vscode+platform.io is so much better !

5-start ratings ,,,, what else to say - even in circuit debugger.

you find 17 homie libraries ... I use this one https://homieiot.github.io/homie-esp8266/docs/3.0.0/quickstart/getting-started/

You find some examples here too ... or:

https://github.com/Christian-Me/esp-christmas-lights/blob/master/src/main.cpp

But you don`t need to any micro to test homie ... just use the homie convention plugin. The homie node enabels node-red to act like a homie device.

You find in the palette manager or latest versions in my repro

https://github.com/Christian-Me/node-red-contrib-homie-convention

happytm commented 4 years ago

Hi @Christian-Me Thank you for your pointers. I have been using ESPEasy for since it's begining for my other projects.But here my goal is to use simpler lightweight firmware I already have. I think your Node-red homie convention node will be most suitable here. But eventually I want to have standalone solution without using Node-Red.

Christian-Me commented 4 years ago

So the homie library is a good starting point ... brings all the homie housekeeping and even OTA updates out of the box ... My christmas light PWM dimmer project ended with 140 lines ... mostly copy paste definitions. Should be not a big deal to join your project. Perhaps (as i tried to explain earlier) Node-Red could be one tool of many ... if everything speaks the same language. One long term project are small 3.5" wall mounted LCDs with an ESP32 (nowadays perhaps ESP32-S2 with 24bit LCD port) speaking homie directly to my devices. So Node-RED is only high level logic and simple switches or dimmer controlls are offloaded to homie-mqtt.

Homie is only the language ... and if you look into the growing list of implementations https://homieiot.github.io/implementations/

But in the end I like the clean, open, and abstract structure ... not defining particular devices. Instead logic and values/parameters and datatypes - something missing in mqtt (everything is a string / buffer) - but added somehow in mqtt 5 ... but it will take ages to replace 3.1. And small messages instead of big JSON structures - much easier to handle on memory restricted micros.

Currently working on a homie-pro node. Exposing everything, Custom extensions and even the convention itself to be customizable. Just pushed the preview version to github ;)

happytm commented 4 years ago

Thank you for your help. It looks like I have lot to learn about homie. I will keep reading and if necessary pop the question here.

Christian-Me commented 4 years ago

Or in the forum that everybody can participate ... Good Night ;)

Christian-Me / remote-device-table

Help needed to implement your code #1