andesse commented 2 years ago

Hue changed the data object on Firmware 1.53.1953188010

This is beta firmware in the moment I’m writing this, but you could also get a problem with the next FW! Turn Auto Update off!

With this flow I am actually not able anymore to get status from light / grouped_light etc.

I try to figure out what happens, even the Id looks strange.

Old, functional firmware

not functional data object, new firmware, big data array

So please don’t Update!

andesse commented 2 years ago

Update: What I found out so far. I figured out the reason why HUE is doing this. Instead of fire a lot of different messages the new firmware combines it to one message with up to 16! data objects. What I figured out so far, one for lights, one for dimming, one for group and a second group (what I have no idea what it is)

In the screenshot you can see my room, Küche, it has the RID shown above as a comment that is used in the change node. At the moment I don’t receive any updates, for this reason: in the new message from the eventstream the correct grouped_light is at data point 3, so all switches need to be changed.

I won’t make any changes in this repository until Saturday evening, cause I still need to figure out what actually changed. If I add a new setup, this one will get a V3 prefix

@FredBlo what do you need for informations, to integrate this into your V2 approach? This is your work, hahah

andesse commented 2 years ago

I made a very ugly workaround that works. As the updated API sends arrays up to 16 objects (in my case,see screenshot) for the reason, that the rid/ ID cycles through the objects, this subflow will fillter up to 20 objects in the array, you can add more if needed. light_event

https://github.com/andesse/hue-clip-api.node-red-flows/blob/main/V3_light_receiver.json

Using this you will get a true / false response again. Everthing else seems to be fine. The problem occurs for lights and group responses. Latency did not change for me, still instant response

andesse commented 2 years ago

I found a better solution that I’ll test now. The data array will get splitted directly after the event flow node. This means that there will be some changes in the switch nodes, cause they won’t search for payload.data.0.id anymore. Instead it will be payload.id

That means I will create a package now for people who need to update their existing flow, and a new version of the main flow.

@FredBlo i will use your flow example to create the new flow for the upcoming update.

andesse commented 2 years ago

New file created https://github.com/andesse/hue-clip-api.node-red-flows/blob/main/V3_receivers.json

andesse commented 2 years ago

New Release

FredBlo commented 2 years ago

@andesse ,

Based on the bridge update you talked about, I spend a 'bit of time' to made a few checks & tests :

HUE Bridge doc

HUE Bridge documentation is quite clear about what occurs here, as stated in eventstream doc :

*"There is a 1 second rate limit on the amount of event containers the Bridge will send. If the same property has changed twice within that timeframe, you only get the last state. If multiple resources have changed within that timeframe, then you will get multiple events grouped in a single container."*

Working as designed / documented from the beginning ... but I presume the new bridge firmware is a bit more aggressive when it has to decide whether it aggregates '.data' or not...

node-red nodes behaviour

Philips HUE events

The Philips HUE events (@yadomi/node-red-contrib-philipshue-events) will 'forward' any message received from the bridge as such. This means it will always be an array of '.data', which can hold 1 to n entries :

-> extract from file 'philipshue-events-config.ts' :

const events = JSON.parse(message.data);
this.emitter.emit("event", events);

HUEMagic (as reference)

The HUEMagic node splits the message it received from the bridge. By doing so, each '.data' of the array will initiate a new flow natively:

-> extract from file 'api.js' :

const messages = JSON.parse(event.data);
for (var i = messages.length - 1; i >= 0; i--)
{
    const message = messages[i];
    [...]
    callback(message.data);
    [...]
}

Possible solution(s)

Node update (Philips HUE events)

The 'philipshue-events-config.ts' could easily be updated to manage data split natively (while keeping every data part received under a single array (data):

const events = JSON.parse(message.data);
for (var i = events.length - 1; i >= 0; i--)
{
    this.emitter.emit("event", [events[i]]);
}

Note : HUEMagic is processing incoming data from last to first, I don't know whether there is a reason for this, but I kept this same logic, not sure it is the way to go...

Flow change

The flow can also be adapted to manage itself splitting '.data' in separated flows. The way you made it in your update is good at doing so. The sole drawback I see is the fact it is not backward compatible (i.e. you have to change your already created (sub)flows since '.data' is moved as such to '.payload'). I would either re-move back payload to payload.data after the split, or use a simple function call which will split as required :

[{"id":"e07e18793ef596bf","type":"comment","z":"68ad1cd9aa26e13b","name":"Converter seperates the array","info":"","x":660,"y":600,"wires":[]},{"id":"b76bc75ac8191065","type":"function","z":"68ad1cd9aa26e13b","name":"1 data per flow","func":"for (var i = 0; i < msg.payload.data.length; i++) {\n    // Create a deep copy of msg object\n    let splitmsg = JSON.parse(JSON.stringify(msg));\n\n    // Reset data of new message to its single data part\n    splitmsg.payload.data = [msg.payload.data[i]];\n    \n    // send using node\n    node.send(splitmsg);\n}\n\nreturn;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":640,"y":560,"wires":[["2a141678adf47667","cc2ef99d4ab569ae"]]},{"id":"5dedd8e42abed89d","type":"philipshue-events","z":"68ad1cd9aa26e13b","bridge":"","x":400,"y":560,"wires":[["b76bc75ac8191065"]]},{"id":"2a141678adf47667","type":"debug","z":"68ad1cd9aa26e13b","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":890,"y":560,"wires":[]},{"id":"cc2ef99d4ab569ae","type":"link out","z":"68ad1cd9aa26e13b","name":"link out 4","mode":"link","links":["0c8f88588b2a2b4f"],"x":835,"y":520,"wires":[]}]

PS : I rather not use a subflow for it because it is normally a single use-case, but maybe you use it for multiple receivers in your system (?)

More globally, I think it would be better / easier to only keep a single valid file with all content in it. In its current state, new users will have more difficult to find their way (do I have to use V2, V3,...), and existing users do not directly see what's new... I would have a v2 'valid' and a v3 beta (which is v2 with your updated approach for now).

andesse commented 2 years ago

@FredBlo I understand the concept in general, BUT on my old FW bridge I get a lot of status messages when I turn off (there came like 5-6 more then on this screenshot)

and the new firmware gives me one message object with an array

I use a subflows with where I change the payload.data to payload and use a split node afterwards. it’s already available for download and works like a charm for me.

And yes, I have two bridges, one for the Livingroom (non beta) and for the rest of the apartment (beta)

i will hit Yadomi up for this reason, I will try to edit the file manually and if it works I’ll ask him to update.

Good point, I will a makeover in the next days and merge everything in one flow. Think there will be need to write the readme more clearly as well.

This was more like an emergency update that everyone can prepare.

FredBlo commented 2 years ago

Your previous explanations were clear about going for multiple messages to a single grouped one.

The function I posted / suggest is actually a splitter which will send 1 separate msg (flow) per .data element in the received array. The result is the same as what you achieved in your subfow, only you keep the initial structure with payload containing '.data' which is an array of 1 single data object (=as before FW upgrade, using msg.payload.data[0]), while the splitter you used modifies the msg structure with msg.payload containing what was before set in msg.payload.data[0]

andesse commented 2 years ago

@FredBlo ok, i went back and used the function node from you now. so nobody need to update the receivers. It is already online for now.

andesse commented 2 years ago

@FredBlo I will merge our two flows tomorrow and update the readme and remove the V2 prefix flow so that there is just one. I like the splitter function node, that should be the best approach for the array. I’ll also add a 3 node flow to create the user token, I just tried it and it works fine and add some more change node examples for actions. Then it should be fine for a while.

I got recently some feedback from different communities, what should be changed in the readme for a better understanding. That’s a thing I want to do as well.

Any suggestions or ideas what need to added as well?

FredBlo commented 2 years ago

I was a bit too fast at analyzing '@yadomi/node-red-contrib-philipshue-events' node-set yesterday :-) (I also looked at code on GitHub, not on deployed node-red, which is indeed a .js with some imports being different, but no change to code itself)

Since I do not have the issue (of .data array) myself, a bit complex to reproduce... But I understood a bit better what -I think- is occurring.

The HUE bridge is already aggregating data 'as documented'... and the current node is also performing a split on incoming events. But the way data are being aggregated by the bridge changed... In previous bridge versions, you received a single eventstream from the bridge being an array of events

[
  {
    "creationtime": "2022-09-06T06:55:43Z",
    "data": [=single array with {content}-],
    "id": "b0ce6895-c31a-41a2-9fc7-392e19947006",
    "type": "update"
  },
  {
    "creationtime": "2022-09-06T06:55:43Z",
    "data": [=single array with {content}-],
    "id": "3ad5ac3f-09e0-4dea-a53f-8f5b7cfa770a",
    "type": "update"
  },
  {
    "creationtime": "2022-09-06T06:55:43Z",
    "data": [=single array with {content}-],
    "id": "0bc30764-ca81-453f-ad13-2cb83c877bb7",
    "type": "update"
  }
]

The yadomi's node correctly split in separated flows (only it is made in 'philipshue-events.js', not in 'philipshue-events-config.js', which I didn't see)

Based, on your info, the new bridge FW also sends a single eventstream container only its globalized at .data level, meaning something like

[
  {
    "creationtime": "2022-09-06T06:55:43Z",
    "data": [
        {data info1},
        {data info2},
        {data info3}
        ],
    "id": "b0ce6895-c31a-41a2-9fc7-392e19947006",
    "type": "update"
  },
]

Conclusions

the added function node should do the job and manage both new and old bridges

If @yadomi/node-red-contrib-philipshue-events needs to be adapted, I made a change in philipshue-events.js file which should do the trick (since it does the same as the added function node, see the inserted loop 'for (var i = 0; i < event.data.length; i++) {')


"use strict";
const nodeInit = (RED) => {
const node = function (config) {
    const status = {
        incoming: () => {
            this.status({ fill: "blue", text: "Incoming..." });
            setTimeout(status.listening, 1000);
        },
        connected: () => {
            this.status({ fill: "green", text: "Connected" });
            setTimeout(status.listening, 2000);
        },
        disconected: () => {
            this.status({ fill: "red", text: "Not connected" });
        },
        listening: () => {
            this.status({ fill: "green", text: "Listening" });
        },
        error: () => {
            this.status({ fill: "red", text: "Error" });
        },
    };
    RED.nodes.createNode(this, config);
    status.disconected();
    this.bridge = RED.nodes.getNode(config.bridge);
    if (!this.bridge)
        return;
    status.listening();
    this.bridge.emitter.on("event", (events) => {
        status.incoming();
        for (const event of events) {
          for (var i = 0; i < event.data.length; i++) {
            // Create a deep copy of msg object
            let splitevent = JSON.parse(JSON.stringify(event));
            // Reset data of new message to its single data part
            splitevent.data = [event.data[i]];
            // send using node
            this.send({ payload: splitevent });
           }
         }
    });
    this.bridge.emitter.on("error", (error) => {
        status.error();
        this.send({ payload: error });
    });
    this.bridge.emitter.on("onopen", status.connected);
};
RED.nodes.registerType("philipshue-events", node);
};
module.exports = nodeInit;


- not sure this is a wanted behaviour from HUE actually, since change is quite important for all already developed connectors... Maybe post about this in their dev / support if you have any contact ?

andesse commented 2 years ago

@FredBlo hi, i reached out to the developers, waiting for reply. I really think that your function spllitter does a great job. In my opinion we should use that one. I started to merge the flows. Here is what i did so far, i want it as easy as possible to understand.

As I had never issues with the http request nodes, i added your approach as an alternative, cause I think that without the limiter the easy http request flow might have a little bit better performance. What do you think?

andesse commented 2 years ago

@FredBlo Forgot to mention: as some users have multiple bridges, i think its more easy to add the data manually to the nodes, instead of setting the data with a Inject/change node. I also removed everthing with a link call, cause some users are not using the most recent NodeRed Version. That would cause a problem when importing.

FredBlo commented 2 years ago

Hi @andesse

Happy to see you merged it as a new version :-)

I personally prefer having subflows only when they have added value (i.e. used multiple times and managed centrally), and I also like exploiting functionalities of latest versions when they work great, but I understand your point about supporting larger scope than node-red v3+.

Anyway, I just gave you some ideas / things I updated to make it work better for me, you decide how you further want this flow to keep evolving :-)

andesse commented 2 years ago

@FredBlo i am very happy to have someone that helps with some code knowledge. I just really have basic knowledge when it comes to functions, that’s the reason I use often several nodes and subflow them. :)

I hope that the repository is good for some time now.

There is some contrib very popular in Germany, it’s called RedMatic (I was using it as well). I am pretty sure >1000 people use it. It’s a NodeRed Addon that is stuck at v2.x or something, cause the dev disappeared. I switched to a Tinkerboard a while ago just for NodeRed, that’s the reason I removed the link calls.

Greetings Andreas

andesse commented 2 years ago

@FredBlo i just realized that the grouping frame is also a NodeRed 3.x thing? I think I should remove it

FredBlo commented 2 years ago

@andesse ,

So Tinkerboard does not allow you running node-red v3 ? I am using the docker version of node-red, which enables to keep latest version runnign very easily (and also having multiple versions / dockers running in parallel for test & debug :-P )

About groups, as far as I remember, I always used such. It certainly existed already in 2.0.x (I checked in my flow backups from then) and maybe also in latest 1.x, but less sure about this).

Anyway, the way node-red manages groups is also backward compatible : when importing JSON flow, group nodes will be ignored but all its content will be imported (since there is no JSON imbrication used for these; a group as an ID and is set on a tab (z index), any node is in a tab (z index) and optionally in a group (g index)).

andesse commented 2 years ago

@FredBlo Hi, the Tinkerboard can run all Versions. I think it’s the most decent SBC available, it comes with an emcc. My thoughts were again users running old Versions. The flow should be simple as possible.

When you’re using it that long I trust what you say. I’m using node Red since version 2.?

Very impressed by your knowledge, haha

Andreas

andesse / HUE-CLIP-API.Node-RED-Flows

ALERT! DO NOT UPDATE YOUR BRIDGE!! #25

HUE Bridge doc

node-red nodes behaviour

Philips HUE events

HUEMagic (as reference)

Possible solution(s)

Node update (Philips HUE events)

Flow change