Webhooks for state changed event listener

[undjike]

Hi @skydiver, I don't know what is the process in Ewelink backend to handle state changes on devices, but I have a suggestion regarding all the problems encountered with Web Sockets.

Why not make use of Webhooks to notify users when any of their devices' state changed ?

[ttz642]

If the users app calls connection.openWebSocket() it can get all state changes.

[undjike]

Yes, I know that. But I've seen a lot of issues on this way of handling changes.

WebSocket auto-closes after some time, an exception is thrown on opening websocket and so on... That's way I suggested usage of webhooks.

[ttz642]

Yes, I know that. But I've seen a lot of issues on this way of handling changes.

WebSocket auto-closes after some time, an exception is thrown on opening websocket and so on... That's way I suggested usage of webhooks.

I had a lot of issues with that also, so I just started listening to the traffic generated by the devices, kinda brute force but reliable, ie:

sudo tcpdump -l -n -i enp3s0 dst 224.0.0.251 and not host 192.168.1.1

This produces output like:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:09:26.012193 IP 192.168.1.33.5353 > 224.0.0.251.5353: 0 [3q] [1au] PTR (QM)? _companion-link._tcp.local. PTR (QM)? _homekit._tcp.local. PTR (QM)? _sleep-proxy._udp.local. (112)
14:09:29.021760 IP 192.168.1.33.5353 > 224.0.0.251.5353: 0 [3q] [1au] PTR (QM)? _companion-link._tcp.local. PTR (QM)? _homekit._tcp.local. PTR (QM)? _sleep-proxy._udp.local. (112)
14:09:33.007668 IP 192.168.1.101.5353 > 224.0.0.251.5353: 0*- [0q] 4/0/0 PTR eWeLink_1000xxxxxx._ewelink._tcp.local., (Cache flush) TXT "txtvers=1" "id=1000xxxxxx" "type=plug" "apivers=1" "seq=32" "encrypt=true" "iv=NzgwMTkzMjM2ODE3MjE0Mg==" "data1=TiRo0HoKbtJdl2dZ4YAU0PWk4ORC8oR7ekXixc58DQvcM6P0S25tXHNbuTK4olMl1fvJLf31pHfw4oFBLZ4AL1bvS881APsLwizw9c0QcXNyCaDKIucPUy1nkDxFV0yOGYu4DqixGuvsdO8l6wAzg7ZdujYd9nCCZH9p17Ewp1VldbW1xTc8hCvz0LUYZXSD", (Cache flush) SRV eWeLink_10006c40ec.local.:8081 0 0, (Cache flush) A 192.168.1.101 (489)
14:09:33.510860 IP 192.168.1.101.5353 > 224.0.0.251.5353: 0*- [0q] 4/0/0 PTR eWeLink_1000xxxxxx._ewelink._tcp.local., (Cache flush) TXT "txtvers=1" "id=1000xxxxxx" "type=plug" "apivers=1" "seq=32" "encrypt=true" "iv=NzgwMTkzMjM2ODE3MjE0Mg==" "data1=TiRo0HoKbtJdl2dZ4YAU0PWk4ORC8oR7ekXixc58DQvcM6P0S25tXHNbuTK4olMl1fvJLf31pHfw4oFBLZ4AL1bvS881APsLwizw9c0QcXNyCaDKIucPUy1nkDxFV0yOGYu4DqixGuvsdO8l6wAzg7ZdujYd9nCCZH9p17Ewp1VldbW1xTc8hCvz0LUYZXSD", (Cache flush) SRV eWeLink_10006c40ec.local.:8081 0 0, (Cache flush) A 192.168.1.101 (489)
14:09:34.009437 IP 192.168.1.101.5353 > 224.0.0.251.5353: 0*- [0q] 4/0/0 PTR eWeLink_1000xxxxxx._ewelink._tcp.local., (Cache flush) TXT "txtvers=1" "id=1000xxxxxx" "type=plug" "apivers=1" "seq=32" "encrypt=true" "iv=NzgwMTkzMjM2ODE3MjE0Mg==" "data1=TiRo0HoKbtJdl2dZ4YAU0PWk4ORC8oR7ekXixc58DQvcM6P0S25tXHNbuTK4olMl1fvJLf31pHfw4oFBLZ4AL1bvS881APsLwizw9c0QcXNyCaDKIucPUy1nkDxFV0yOGYu4DqixGuvsdO8l6wAzg7ZdujYd9nCCZH9p17Ewp1VldbW1xTc8hCvz0LUYZXSD", (Cache flush) SRV eWeLink_10006c40ec.local.:8081 0 0, (Cache flush) A 192.168.1.101 (489)
14:09:34.510108 IP 192.168.1.101.5353 > 224.0.0.251.5353: 0*- [0q] 4/0/0 PTR eWeLink_1000xxxxxx._ewelink._tcp.local., (Cache flush) TXT "txtvers=1" "id=1000xxxxxx" "type=plug" "apivers=1" "seq=32" "encrypt=true" "iv=NzgwMTkzMjM2ODE3MjE0Mg==" "data1=TiRo0HoKbtJdl2dZ4YAU0PWk4ORC8oR7ekXixc58DQvcM6P0S25tXHNbuTK4olMl1fvJLf31pHfw4oFBLZ4AL1bvS881APsLwizw9c0QcXNyCaDKIucPUy1nkDxFV0yOGYu4DqixGuvsdO8l6wAzg7ZdujYd9nCCZH9p17Ewp1VldbW1xTc8hCvz0LUYZXSD", (Cache flush) SRV eWeLink_10006c40ec.local.:8081 0 0, (Cache flush) A 192.168.1.101 (489)
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

I also filter out duplicates, looks like the devices send the message multiple times to make sure they get through.

Simply decode the message and it shows you the current state of the device.

I have NEVER missed a state change since using this method.

btw I now only using the ewelink-api to get device definitions and control and monitoring is using zeroconf / mDNS.

[AMalick]

I can confirm that after implementing this code change to my node red setup I too have not had any websocket timeout issues

[ttz642]

This was my first hack a few months ago.I have re-written this completely in node.js with idea from another project. I'll post within the week and credit them for their contribution.

btw: I'm now at the stage where I only use the cloud to configure the devices and update them, from then on I'm local only. I do only use light switches and plug sockets.

[AMalick]

Cool.. I only use this for door/window contact sensors via node red. So had to patch and glue together this fix to manually update your V2.0.0 Node Ned evenListner node to make it work.

Any plans to release an updated Node Red version ?

BTW... Thank you for your awesome work on this.. saved me a huge amount of effort. 👍

[AMalick]

btw: I'm now at the stage where I only use the cloud to configure the devices and update them, from then on I'm local only. I do only use light switches and plug sockets.

Its awesome that your local only.. I was planning to try and get around to that but the Door and Window sensors cant really be turned into DIY mode like the on/off relay stuff unfortunately.

[ttz642]

Originally I used tcpdump and parsed the output and posted to my handler.

There's so many node modules and finding the right one to monitor the traffic was helped when I came across homebridge-ewelink. By using the node-dns-sd I was able to wrap all the monitoring and filtering of duplicate packages in node.js.

#!/usr/bin/node

const dns       = require('node-dns-sd');
const axios     = require('axios');

var lastChange  = {};
var blanking;

async function startMonitor () {
    dns.ondata = packet => {
        try{
            if(packet.header.answers == 4){
                var id = packet.answers[1].rdata.id;
                var iv = packet.answers[1].rdata.iv;
                if(lastChange[id] != iv){   //ignore duplicates, if the iv hasn't changed it's a duplicate
                var data1 = packet.answers[1].rdata.data1;
                    var payload = {
                        id,
                        iv,
                        data : data1
                    };
                    //
                    axios.post('http://127.0.0.1:5555/zeroconf', payload)
                    .then((res) => {
                    }).catch((err) => {
                        console.error(new Date().toISOString(),' | error:',err);
                    });
                    lastChange[id] = iv;
                    clearTimeout(blanking);
                    blanking = setTimeout(function() {
                        lastChange = {};
                    }, 3000);
                };
            };
        }catch{};
    };
    dns.startMonitoring();
};

async function main () {
    console.log(new Date().toISOString(),'| Starting mDNS/DNS-SD monitor...');
    await startMonitor();
};

main();

It runs standalone and posts the captured packets which are then decoded and used.

[ttz642]

I was planning to try and get around to that but the Door and Window sensors cant really be turned into DIY mode like the on/off relay stuff unfortunately.

Are you sure ?

If they update the cloud like the switches they just send out a state change and it should be picked up by the monitor code as it's broadcast.

If you need to prompt the devices to output their status I simply scan the network for zeroconf devices and they just output their state as if they were sent a query:

avahi-browse -t -d local _ewelink._tcp --resolve -tp > /dev/null

Wrapped in my web server posting to url://status will scan the local network for devices:

/*-------------------------------------------------------------------------------------------------------------------*/
app.post('/status', function(req,res){
    (async () => {
        //  avahi-browse -t -d local _ewelink._tcp --resolve -tp
        var cmd = 'avahi-browse -t -d local _ewelink._tcp --resolve -tp > /dev/null';
        var status = exec(cmd, (error, stdout, stderr) => {
            if (error) {
                console.error(new Date().toISOString(),`| error: ${error.message}`);
                return;
            }
            if (stderr) {
                console.error(new Date().toISOString(),`| stderr: ${stderr}`);
                return;
            }
            //console.log(new Date().toISOString(),`|stdout: ${stdout}`);
        });
        console.log(new Date().toISOString(),'| Fetch device(s) status...');
        res.end();
    })();
});

I'm running on ubuntu / linux and I just call the avahi-browse to scan for devices, I'm sure there's a node module and i'll incorporate that at some point. There's no need to do anything with the results as the monitor will pick the broadcast messages up.

So after I've scanned for devices and have their state I just monitor for state changes, not missed any !

[rafagil]

@ttz642 Hi, thank you very much for sharing this avahi command to get the device status. I tried here and I was able to get the device status.

However I'm not able to decrypt the data. I tried using the "decriptionData" function from ewelink.js but I only was able to get the start of the json for data3 and I get empty responses for data1 and data2.

Is there anything else needed to decrypt the data?

[ttz642]

@ttz642 Hi, thank you very much for sharing this avahi command to get the device status. I tried here and I was able to get the device status.

However I'm not able to decrypt the data. I tried using the "decriptionData" function from ewelink.js but I only was able to get the start of the json for data3 and I get empty responses for data1 and data2.

Is there anything else needed to decrypt the data?

You have to use the devices info that you cached, I've created a program to get the device data and store in a file and then load that. I don't cache everything about the device, just the essential bits, also I have added a label against a device description, so instead of calling a light switch "kitchen", I call it "L1:kitchen" then I can refer to the light switch using the device id or the label, ie L1

#!/usr/bin/node

//const cacheFileName      = 'conf/lan-cache.json';

const fs   = require('fs');
const ltbl = JSON.parse(fs.readFileSync("conf/ltbl-config.json"), 'utf8');

const dns         = require('node-dns-sd');
const ewelink  = require('ewelink-api');

var devIPmap = {};
var deviceInfo = {};

async function getHosts () {    //homebridge-ewelink
    const res = await dns.discover({ name: '_ewelink._tcp.local' });
    //console.log('res:',res);
    res.forEach(device => {
        const deviceId = device.fqdn.replace('._ewelink._tcp.local', '').replace('eWeLink_', '');
        const ipAddr = device.packet.address;
        devIPmap[deviceId] = ipAddr;
        console.log('Found device:\t'+deviceId);
    });
    return this.deviceMap;
}

function account() {
    return {
        email: ltbl.ewelink.email,
        password: ltbl.ewelink.password,
        region: ltbl.ewelink.region
    };
}

async function main () {
    console.log('Starting...');
    let deviceMap = await getHosts();
    console.log('device Ip\'s',devIPmap);
    var connection = new ewelink(account());// create connection to ITLEAD server
    console.log("\nCreate devices cache...");
    const devices = await connection.getDevices();
    Object.keys(devices).sort().forEach(function(key) {
        var dev = devices[key];
        if((dev.deviceid != null)&&(dev.location != null)){
            var id = dev.name.split(':'); // split name into label & name
            deviceInfo[id[0]] = {
                type : 'label',
                label : id[0],
                deviceid : dev.deviceid,
                name : dev.name,
                location : id[1],
                IP: devIPmap[dev.deviceid],
                hostname : 'eWelink_'+dev.deviceid+'.local',
                apikey : dev.apikey,
                devicekey : dev.devicekey
            };
            deviceInfo[dev.deviceid] = {
                type : 'deviceid',
                deviceid : dev.deviceid,
                label : id[0],
                name : dev.name,
                location : id[1],
                IP: devIPmap[dev.deviceid],
                hostname : 'eWelink_'+dev.deviceid+'.local',
                apikey : dev.apikey,
                devicekey : dev.devicekey
            };
        }
    });
    console.log('LAN devices::',deviceInfo);
    var cacheFileName = ltbl.cache;
    try {
        fs.writeFileSync(cacheFileName, JSON.stringify(deviceInfo, null, 2), 'utf8');
        return { status: 'ok', file: cacheFileName };
    } catch (e) {
        console.log('An error occured while writing JSON Object to File:',cacheFileName);
        return { error: e.toString() };
    }
}

main();

So I get an array of device info searchable by label or deviceid, eg one light switch:

  "L6": {
    "type": "label",
    "deviceid": "1000xxxxxx",
    "label": "L6",
    "name": "L6:sonoff",
    "location": "sonoff",
    "IP": "192.168.1.106",
    "hostname": "eWelink_1000xxxxxx.local",
    "apikey":      "66666666-2222-bbbb-aaaa-cccccccccccc",
    "devicekey": "66666666-2222-bbbb-aaaa-cccccccccccc"
  },
  "1000xxxxxx": {
    "type": "deviceid",
    "deviceid": "10008ba9c9",
    "label": "L6",
    "name": "L6:sonoff",
    "location": "sonoff",
    "IP": "192.168.1.106",
    "hostname": "eWelink_1000xxxxxx.local",
    "apikey":      "66666666-2222-bbbb-aaaa-cccccccccccc",
    "devicekey": "66666666-2222-bbbb-aaaa-cccccccccccc"
  }

So search for L6 or 1000xxxxxx, only have small number of devices so easier / faster to duplicate than have multiple arrays or indirect lookup.

#!/usr/bin/node
const confFileName = 'conf/lan-cache.json';

var DEBUG = 1;
const PORT_WWW=5555;
const   dashboard_url = 'http://127.0.0.1:8888/';

var S               = require('string');        //https://www.npmjs.com/package/string
var fs              = require('fs');
const path          = require('path');
const axios         = require('axios');
const {exec, execSync} = require('child_process');

var express         = require("express");
var bodyParser      = require("body-parser");
const app           = express();
app.use(bodyParser.urlencoded({ extended: false }));//originally true
app.use(bodyParser.json());

var devs;
var lastresponse = {};

//============================================================================================
//  const { nonce, timestamp } = require('../node_modules/ewelink-api/src/helpers/utilities');
//--------------------------------------------------------------------------------------------
const nonce = Math.random()
  .toString(36)
  .slice(5);
const timestamp = Math.floor(new Date() / 1000);
//============================================================================================
//  const deviceControl = require('../node_modules/ewelink-api/src/mixins/deviceControl');
//--------------------------------------------------------------------------------------------
const crypto = require('crypto');
const CryptoJS = require('crypto-js');
const random = require('random');
const makeAuthorizationSign = (APP_SECRET, body) =>
  crypto
    .createHmac('sha256', APP_SECRET)
    .update(JSON.stringify(body))
    .digest('base64');
    const create16Uiid = () => {
        let result = '';
        for (let i = 0; i < 16; i += 1) {
            result += random.int(0, 9);
        }
    return result;
};
const encryptionBase64 = t =>
CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(t));
const encryptationData = (data, key) => {
    const encryptedMessage = {};
    const uid = create16Uiid();
    const iv = encryptionBase64(uid);
    const code = CryptoJS.AES.encrypt(data, CryptoJS.MD5(key), {
      iv: CryptoJS.enc.Utf8.parse(uid),
      mode: CryptoJS.mode.CBC,
      padding: CryptoJS.pad.Pkcs7,
    });
    encryptedMessage.uid = uid;
    encryptedMessage.iv = iv;
    encryptedMessage.data = code.ciphertext.toString(CryptoJS.enc.Base64);
    return encryptedMessage;
};
const decryptionBase64 = t =>
CryptoJS.enc.Base64.parse(t).toString(CryptoJS.enc.Utf8);
const decryptionData = (data, key, iv) => {
    const iv64 = decryptionBase64(iv);
    const code = CryptoJS.AES.decrypt(data, CryptoJS.MD5(key), {
    iv: CryptoJS.enc.Utf8.parse(iv64),
    mode: CryptoJS.mode.CBC,
    padding: CryptoJS.pad.Pkcs7,
  });
  return code.toString(CryptoJS.enc.Utf8);
};
//============================================================================================

function post(url,json){
    axios.post(url, json)
    .then((res) => {
        //console.log(`Status: ${res.status}`);
        //console.log('Body: ', res.data);
    }).catch((err) => {
        console.error(new Date().toISOString(),' | error:',err);
    });
}

/* process local lan switch change */
//  http POST 127.0.0.1:5555/zeroconf id="1000xxxxxx" iv="MTU3NDU4ODI2NDY4NTg0MQ==" data="fQxsTMoZtumKLsUUJScAufPABrDuh3ZvG/YkLbpf63kgdGEVIfMyMyKxGf4q5yS13kZQK5xTI27daasst+vpMAuSgKqvLrHD/y49rFYT2WY+wC0ln4HhRSRVdVND0iisYVCP9d4Kntm4orUC00zwCBUGZZo930GIYYcA63e1U4yXId10bNzBhdFroIHly5zV"
app.post('/zeroconf', function(req,res){
    var label = devs[req.body.id].label;
    //console.log('packet:',req.body);
    var iv = req.body.iv;
    var data = req.body.data;
    var key = devs[label].devicekey;
    var message = decryptionData (data, key, iv);
    var returned = JSON.parse(message);
    //console.log('decrypted payload:',returned);
    post(dashboard_url+'switch/'+devs[label].deviceid+'/'+returned.switch,'');
    console.log(new Date().toISOString(),'|',label,'changed state to:',returned.switch);
    lastresponse[label] = iv;
    res.end('');
});

// switch : dev_state
function makePayload(devid, dev_state) {
    msgSwitch = {
        deviceid : '',
        switch : dev_state
    };
    encryptedMessage = encryptationData(JSON.stringify(msgSwitch), devs[devid].devicekey);
    var payload = {
        sequence : Math.floor(timestamp * 1000).toString(),
        deviceid : devs[devid].deviceid,
        selfApikey : devs[devid].apikey,
        iv : encryptedMessage.iv,
        encrypt : true ,
        data : encryptedMessage.data
    };
//    console.log("command:",JSON.stringify(msgSwitch));
//    console.log("payload",JSON.stringify(payload));
    return payload;
}

function sonoffSetSwitch(label,payload,method) {
    //console.log('Switch:',label,'IP:','http://'+devs[label].IP + ':8081/zeroconf/switch','with payload:',payload)
    axios.post('http://'+devs[label].IP + ':8081/zeroconf/switch', payload)
    .then((res) => {
        //console.log(`Status: ${res.status}`);
        //console.log('Body: ', res.data);
    }).catch((err) => {
        console.error(new Date().toISOString(),' | error:',err);
    });
}

/* turn device "on" or" off" */
app.post('/onoff/:id/:switch(on|off)', function(req,res){
    var state = req.params.switch;
    var id = req.params.id;
    var label = devs[id].label;
    console.log(new Date().toISOString(),'| Change state of',label,'to',state);
    var payload =  makePayload(label,state);
    sonoffSetSwitch(label,payload,1);
    res.end(JSON.stringify(state));
});

devs = JSON.parse(fs.readFileSync(confFileName), 'utf8');
//console.log('devs::',devs);

//Start listener
app.listen(PORT_WWW,function(){
    console.log("Started lanZeroconf at: "+ new Date() +" on PORT " + PORT_WWW);
});

I operate totally offline, only use ewelink app to add devices or update firmware, use lan mode and cached device data, very reliable, never miss an event or drop connections.

When I get round to it I'll publish all the code as an app, have dashboard and scheduler.

[rafagil]

@ttz642 Thanks for sharing the entire code! I'll study it and try to replicate on mine here. I'm also operating fully offline using iOS Siri shorcuts, but, since I can't get the current status, things get out of sync every time someone manually switches the light 😅