failed to load smarthome devices: "Cannot convert undefined or null to object"

[mdte123]

Anyone seeing this behaviour: authentication seems to work fine, the alexa speak and sound functions work. But I'm seeing nothing in my list of smarthome devices and all smarthome nodes are now non-functional with ??? in place of the devices that should be there.

I do have this suspicious error in my node-red log:

Jul 14 12:46:35 system Node-RED[14321]: 14 Jul 12:46:35 - [warn] [alexa-remote-account:Alexa Account 2021] failed to load smarthome devices: "Cannot convert undefined or null to object"

I did an upgrade to NR 2.2.2 on my pi yesterday with nodes 14. But it was functioning fine for hours after that. Something has gone wrong in the small hours of the night.

Have also tried configuring a new alexa-remote-account (negotiating the cookie or whatever it does) which results in the same problem.

[ghost]

Same problem here ...

Node red 2.2.2 Applestrudel 5.0.9

[bbindreiter]

Hi! Just to understand the problem, you mean this list, right?

[mdte123]

Yep. That list. It happens when Action or Query is selected.

I also get lots of these errors:

Jul 14 17:19:44 system Node-RED[18524]: 14 Jul 17:19:44 - [error] [alexa-remote-smarthome:XXXXXX] smarthome entity not found: "4e9c36de-XXXXXX"

[mdte123]

So this morning, immediately after a reboot (node red was down all night and the reboot restarted it), I had these messages. Restarting node-red at that point led to the the nodes re-initialising, authenticating and speak functions working, but with an empty device list (as per above).

It's possible that this is some kind of amazon rate limiting thing on the number of device checks that can be done.

Jul 15 09:20:59 system Node-RED[426]: 15 Jul 09:20:59 - [error] [alexa-remote-smarthome:Check state of bedroom TV Plug] error response: Too many requests. Try again in some time.
Jul 15 09:21:00 system Node-RED[426]: 15 Jul 09:21:00 - [error] [alexa-remote-smarthome:f8dcdcaf.97d078] error response: Too many requests. Try again in some time.
Jul 15 09:21:02 system Node-RED[426]: 15 Jul 09:21:02 - [error] [alexa-remote-smarthome:Check if bathroom lights state] error response: Too many requests. Try again in some time.
Jul 15 09:21:14 system Node-RED[426]: 15 Jul 09:21:14 - [error] [alexa-remote-smarthome:log view ON] error response: Too many requests. Try again in some time.
Jul 15 09:21:29 system Node-RED[426]: 15 Jul 09:21:29 - [error] [alexa-remote-smarthome:Check state of bedroom TV Plug] error response: Too many requests. Try again in some time.
Jul 15 09:21:56 system Node-RED[426]: 15 Jul 09:21:56 - [error] [alexa-remote-smarthome:Turn bedroom lights off] error response: Too many requests. Try again in some time.
Jul 15 09:22:09 system Node-RED[426]: 15 Jul 09:22:09 - [error] [alexa-remote-smarthome:log view ON] error response: Too many requests. Try again in some time.
Jul 15 09:22:27 system Node-RED[426]: 15 Jul 09:22:27 - [error] [alexa-remote-smarthome:58a37fe1.344048] error response: Too many requests. Try again in some time.
Jul 15 09:22:29 system Node-RED[426]: 15 Jul 09:22:29 - [error] [alexa-remote-smarthome:Check state of bedroom TV Plug] error response: Too many requests. Try again in some time.
Jul 15 09:22:47 system Node-RED[426]: 15 Jul 09:22:47 - [error] [alexa-remote-smarthome:58a37fe1.344048] error response: Too many requests. Try again in some time.
Jul 15 09:22:55 system Node-RED[426]: 15 Jul 09:22:55 - [error] [alexa-remote-smarthome:State KitchenLights Group] error response: Too many requests. Try again in some time.
Jul 15 09:22:59 system Node-RED[426]: 15 Jul 09:22:59 - [error] [alexa-remote-smarthome:Check state of bedroom TV Plug] error response: Too many requests. Try again in some time.

[mayankraichura]

I can concur that this happening to me as well. When I enabled TRACE flag, I'm getting lots of Too many requests. Try again in some time. error which probably results in empty response and therefore "Cannot convert undefined or null to object".

[mdte123]

@mayankraichura : Forgive my ignorance, but how do you enable the TRACE flag on these nodes?

[stingrayuk]

I have the same issue with the same error messages as reported by others. I was running 5.0.9 and have upgraded to 5.0.10 but see the same issue.

[mdte123]

I've managed to disable all my alexa nodes apart from three testing ones and the limiting still appears to be present. I will wait a bit to see if it comes back.

One of the nodes has this config (which may be easier than looking at logs) to directly check this:

Strangely, the node reports success when this happens.

[mayankraichura]

@mayankraichura : Forgive my ignorance, but how do you enable the TRACE flag on these nodes?

You can pass -D logging.console.level=trace as start option.

OR

you can add/modify an entry for logging.console.level in settings.js

logging: {
    // Console logging
    console: {
        level: "trace",
        metrics: false,
        audit: false
    }

Reference: Node-Red Logging

[driagi]

I've the same problem!

[mayankraichura]

So the issue seems to be resolved at this moment. Probably, it was an outdated cookie or something that changed either at Amazon or in the alexa-remote2 upstream code.

All I had to do was delete the json file defined in Alexa Account node and re-deploy. Once reauthorized, the device list started working.

[mdte123]

I tried that again. Didn't work for me. Though that has been the problem in the past. It would mean that the speak functions and Get Groups etc would also stop working. This is a specific problem with the Get Devices functionality. Looking at my test flows, I've managed to get the device list once in the last hour or something. The rest of the time it has been "Too many requests".

[mayankraichura]

@mdte123 Same here. It's a hit or miss. It works randomly but stops working on next restart. From the sequence of errors, it seems that the smarthome system fails before the account is initialized. Probably because the account init is taking longer than expected?

I can see in the upstream alexa-remote2 that in 5.4.0, they have Increase timeouts for getting smart home device data in changelog. May be this has something to do with the delay in init or something. node-red-contrib-alexa-remote2-applestrudel is using 5.3.0.

@bbindreiter Can we bump the alexa-remote2 to 5.4.0 to try and see if that resolves the issue?

I don't have working development environment to test my theory but will try to setup one and give it a shot.

[mayankraichura]

Forgive me for spamming this feed but I'm just as much trying to log all the possibilitiese to assist you guys as much as I can.

The this node is using ^5.3.0 for alexa-remote2 which means that it's already at the most recent version which is at 5.6.0. So I'm out of ideas now.

PS: When I use "Fresh Authentication" to initialize the nodes, it works. But fails it used with Initialize using cookie and the sequence of errors still points at either timeout or out of sequence execution of events.

[mdte123]

@mayankraichura : Don't worry, Thanks for helping!

But am curious. I have applestrudel 5.0.9 which lists remote2 as 4.1.9 in my package.json (which just came from a node web gui install:

{
  "_from": "node-red-contrib-alexa-remote2-applestrudel@5.0.9",
  "_id": "node-red-contrib-alexa-remote2-applestrudel@5.0.9",
 <SNIP>
  "dependencies": {
    "alexa-remote2": "^4.1.2"
  },
 <SNIP>
  "version": "5.0.9"
}

I don't really understand node's too much but 'npm ls' bears this dependency relationship out. I dont think I ever explicitly installed alexa-remote2 and I dont have a 5.X.X version there.

In any case, I'm still getting the odd once-per-hour successful (am trying every 10 mins) lookup of the devices. Again there are no problems on the authentication side.

Wild speculation: Since it works once per hour I can only suspect that there isn't too much wrong with the code. Buy maybe amazon have depreciated this URL in their api for devices and are rate limiting the older one back or something. It could be that a newer version of whatever http request path is required. Or it could be that amazon are just rate limiting me because I have a lot of lookups.

My house has time travelled back to the 1990s.

"node --version" is at v 14.20.0. All my account node settings are for the UK.

[mayankraichura]

One more update: If I patiently try to reinitialize using proxy method, once a minute, I am able to update the device list which is hopefully cached until next initialize.

However, getting states is still an issue it gives an error error response: Too many requests. Try again in some time."

[mayankraichura]

Here is the debug log:

15 Jul 22:31:57 - [debug] [alexa-remote-account:9f753fc1a554b347] Alexa-Remote: Sending Request with {"host":"alexa.amazon.in","path":"/api/phoenix?includeRelationships=true&_=1657904517526","method":"GET","timeout":30000,"headers":{}}
15 Jul 22:31:57 - [debug] [alexa-remote-account:9f753fc1a554b347] Alexa-Remote: Response: {"message":"Too many requests. Try again in some time."}
15 Jul 22:31:57 - [warn] [alexa-remote-account:9f753fc1a554b347] failed to load smarthome devices: "Cannot convert undefined or null to object"

[mayankraichura]

@mdte123 Yes. Same here. I was using lookups to query for virtual devices for smart devices that did not have have integrations compatible with Home Assistant. So for me, it was around 8 queries every 10 seconds. However, I've disabled those queries since yesterday and yet I'm getting rate limited.

So if there is quota for that it's either per week/month instead of per hour/day or maybe my account is blacklisted or something.

[mdte123]

Thanks @mayankraichura mines doing one lookup every 15 mins now, will leave it overnight and see if it gets better. If not, I guess we're a bit stuck.

[bbindreiter]

Some hints I found in the dependency changelog:

5.1.0 (2022-07-04) (Apollon77) Detect Rate limit exceeded response and do one automatic request retry 10s later (plus a random part) (Apollon77) Calculate the timeout of querySmarthomeDevices dynamically between 10s and 60s (maximum overrideable by new optional parameter) depending on the number of devices to query

Unfortunately I'm currently a bit busy but will have a look asap.

[bbindreiter]

@mdte123 Same here. It's a hit or miss. It works randomly but stops working on next restart. From the sequence of errors, it seems that the smarthome system fails before the account is initialized. Probably because the account init is taking longer than expected?

I can see in the upstream alexa-remote2 that in 5.4.0, they have Increase timeouts for getting smart home device data in changelog. May be this has something to do with the delay in init or something. node-red-contrib-alexa-remote2-applestrudel is using 5.3.0.

@bbindreiter Can we bump the alexa-remote2 to 5.4.0 to try and see if that resolves the issue?

I don't have working development environment to test my theory but will try to setup one and give it a shot.

Yes, I will update the dependency and push out 5.0.11. On it Update is available!

I have tried to reproduce the issue with the Get -> Devices node. Triggered it a hundred times and oddly enough, I do not get any rate limit errors.

[stingrayuk]

Thank you @bbindreiter, I have installed 5.0.11 and still have the "error response: Too many requests. Try again in some time." when using Get -> devices and therefore have ??? when trying to set a device.

I have disabled my flow and will leave disabled overnight and retry in the morning.

Thank you to all for the efforts in trying to fix this issue.

[stingrayuk]

I have enabled my flow again this morning, but still have the same issue

I have deleted my alexa cookie and can re-initialise fine, but trying a get devices gives the response above.

[mayankraichura]

@bbindreiter I guess its the problem with Alexa rate limiting and probably has nothing to do with your node. I tested out node-red-contrib-alexa-remote2-mattl0 which is similar to yours and still facing same issue.

[bbindreiter]

Thanks! German speaking folks can check out this discussion here going on here as well: https://forum.iobroker.net/topic/56021/alexa2-adapter-3-13-3-14-3-15-3-16-3-17-beta

[stingrayuk]

I also found this that may help: https://aws.amazon.com/premiumsupport/knowledge-center/aws-batch-requests-error/

[mdte123]

I've upgraded to 5.0.11 and confirm that the same behaviour is present. Its doing a get devices lookup about every 15 mins with about a 25% success rate. No change from the 5.0.9 install.

[mdte123]

Another factoid: if on a re-initialisation with the init node it happens to get a device list response, then subsequent smartphone "turn something on" nodes (which now have a list populated) also fail with a "error response: Too many requests. Try again in some time." message. So I think there's rate limiting there too.

[mdte123]

Hi all,

I think you're gonna like this... they are rate limiting on the User-Agent (which is an option in the account configuration settings). I just set that to be the same as my browser and bingo... 100% working.

Happy days.

[bbindreiter]

I'll check what's being sent as fallback value later today or tomorrow and will set this to something more recent.

What did you set it to exactly @mdte123? Thank you!

[mdte123]

The user agent of my safari browser on my MacBook pro (but probably any modern browser user agent will do):

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15

I suspect that they need to not rate limit this so as their developers can run test code. However, it would be good to figure out what the User-Agent of an alexa device is. Then they'd have no easy way of distinguishing messages from their own devices. Its probably a sensible idea from their point of view to reduce processing from garbage internet bots and crawlers.

Its also possible that the user-agent is missing from the web requests that alexa-remote2 or applestrudel send (haven't checked), if so then that would look a bit suspect and should be rate limited.

Thanks to everyone for their help! I have a functioning house again. Now all I need is for someone to implement a routine to turn the sun down a bit in the UK any chance? ;-)

[stingrayuk]

@mdte123, thank you for your excellent discovery, I have used the following browser agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36

and all is working again.

Thanks to everyone for your help in sorting this out. All is now working again :-)

[obaldius]

The user agent of my safari browser on my MacBook pro (but probably any modern browser user agent will do):

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15

I suspect that they need to not rate limit this so as their developers can run test code. However, it would be good to figure out what the User-Agent of an alexa device is. Then they'd have no easy way of distinguishing messages from their own devices. Its probably a sensible idea from their point of view to reduce processing from garbage internet bots and crawlers.

Its also possible that the user-agent is missing from the web requests that alexa-remote2 or applestrudel send (haven't checked), if so then that would look a bit suspect and should be rate limited.

Thanks to everyone for their help! I have a functioning house again. Now all I need is for someone to implement a routine to turn the sun down a bit in the UK any chance? ;-)

Excuse me for the stupid question but could you elaborate a bit what to do in order to get this working again please? Where and how should I set the browser agent? thanks

[bbindreiter]

@obaldius Alexa Account Node Settings on the bottom:

[Apollon77]

The rate limit issues should be better already now and next version of alexa-remote will optimize stuff there (yes the user agent is one part of it).

Please still be VERY CAREFUL which smart home data you query how often (@bbindreiter please ad a big disclaimer/notice/warning somehwere!) because potentially each call produces traffic and costs also on the Skill developer side ...

[mayankraichura]

@Apollon77 Thanks for the suggestion. Personally, I used it to query POWERSTATE on 6 devices every 10 seconds to report it back to Home Assistant. Howevery, I have disabled regular polling due to rate limit errors. I guess that is what is causing the rate limiting. Is there a way that Alexa would push updates via WS or something? That way we would not need to query the device states.

[bbindreiter]

What devices are you polling?

[Apollon77]

In fact such things like "query every 10s" are causes for such limitations. it might be ok if one user does that ... but the more it are the bigger the problem will be. So please be careful in general.

For now there are no available "Push" options that we know and in fact the API used here is the one used by the Alexa App itself... and so this is not really meaned to be "continously queried".

The newst version 5.7.0 will add some options to optimize querying ... @bbindreiter if you like more insights contact me via email please (iobroker@fischer-ka.de)

[mdte123]

Agree, I find the best way to do this is not to poll. Rather, do the lookups when you need them based on some kind of inbound signal (like a motion sensor or something). motion sensor -> do lighting state lookups -> turn lights on/off. This is more reliable and avoids race conditions where the polled lookup isn't the same as what actually is.

[mayankraichura]

Thanks @Apollon77 @bbindreiter @mdte123 for your responses.

Yes, I quickly realized this issue and stopped the automated polling since last 2 days and planning around to find a way to notify states in home-assistant. It only becomes an issue when the devices are trigged out of the HA/NR, eg apps or physical switches, since the device states would be out of sync.

Anyways, thanks for quick replies from all of you and since it's already clear that the issue was with Amazon and not the lib, I guess it would be safe to close this issue at the author's convenience.

[stingrayuk]

Thanks to all for efforts in fixing this and I have had no issues since changing the user agent, but reading through @Apollon77 excellent work to release alexa-remote2 at 5.7.0 @bbindreiter is it sensible to bump the alexa-remote2 dependency to 5.7.0 to take advantage of all the changes?

[bbindreiter]

Yes, already in the works.

[bbindreiter]

5.0.18 is available. It updates alexa-remote2 to 5.7.3 .

Thanks everyone!