Google Local Fulfillment broken in 2022.6.0b1

[pdobrien3]

I have been trying to get google local fulfillment to work on the new beta. I uploaded app,js version 2.1.5, can see my google minis under chrome://inspect#devices. When I inspect the device, I can see Ready, App version: 2.1.5 in the top line of the console. I have tailed the NGINX logs and can see the request getting to NGINX but then I get this:

[27/May/2022:17:39:11 +0000] "GET /api/google_assistant HTTP/1.1" 405 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36" [27/May/2022:17:39:13 +0000] "GET /service_worker.js HTTP/1.1" 200 56579 "https://hass.my_domain.com/service_worker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"

According to the docs, a 405 error means that Google Assistant is working yet when I tell google to turn on a light, I get...sorry it looks like the office light is not available right now.

google home works. Sync devices works. TTS works. Only thing that doesn't work is voice requests to HAss

In the google cloud platform log explorer I can see sync requests and when I restart the minis I can see this:

2022-05-27 13:07:58.199 EDT {"appVersion":"2.1.5", "intent":"IDENTIFY", "requestId":"Redacted", "severity":"INFO", "timestamp":"2022-05-27T17:07:57.96Z"}

{ insertId: "Redacted" jsonPayload: { appVersion: "2.1.5" intent: "IDENTIFY" requestId: "Redacted" severity: "INFO" timestamp: "2022-05-27T17:07:57.96Z" } logName: "projects/homeassistant-ab4d3/logs/assistant_smarthome%2Fintent_error_logs" receiveTimestamp: "2022-05-27T17:07:58.199806055Z" resource: { labels: { project_id: "homeassistant-ab4d3" } type: "assistant_action_project" } severity: "INFO" timestamp: "2022-05-27T17:07:58.199806055Z" }

Willing to try anything at this stage!!!

[pdobrien3]

NGINX Access logs. Not sure if this is a legit attempt to access something or not but if it is legit, it is making an http call to my domain which should be https?

[28/May/2022:06:39:44 +0000] "GET / HTTP/1.1" 444 0 "-" "Mozilla/5.0 (compatible; Google-Test;)" [28/May/2022:06:46:16 +0000] "GET / HTTP/1.1" 444 0 "http://.com" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36"

also, this:

[27/May/2022:17:15:24 +0000] "GET /favicon.ico HTTP/1.1" 404 14 "https://hass..com/api/google_assistant" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"

[pdobrien3]

after enabling debug logs in the actions console, when i do a voice command, the google cloud platform logs look like they query the state of everything exposed to google instead of attempting a state change of the device.

[pdobrien3]

reverting to version 1.1.0 of app.js fixed my issue.

[balloob]

You need to debug the local SDK when it runs on your Google Home. See https://github.com/NabuCasa/home-assistant-google-assistant-local-sdk#chrome-debugging. Once you click inspect, it will show you what is actually going on and show errors.

[pdobrien3]

Do I need to do this section: https://github.com/NabuCasa/home-assistant-google-assistant-local-sdk#local-sdk-dev-env

And do I need zeroconf and cloud in my config.yaml? I have an older install and do not have default_config.

[balloob]

You don't need a local dev env. You do need zeroconf.

[pdobrien3]

i am not sure what to look for in the logs. i have screen shots. is there anything sensitive in the inspect logs. i basically only have 5 warning entries and the entry that lists version 2.1.5. when i am on the earlier version there is multiple entries in the inspect logs. There are no errors. The logs show all my entities exposed to google being discovered but then that’s about it.

[calisro]

I am seeing differences after upgrading to 2.1.5.

I see it intermittantly working where it used to work consistently. I've checked all the debugging info and i can provide ( I am not sure if there is any private information in those inspect logs though!)

This is what I am seeing:

As you can see its intermittant. I can use the same light and the same command and it will work part of the time and still switch to cloud the other. I don't see errors in the device inspect logs. I see it sends the info to HA just fine:

[calisro]

I should note that when it uses partner_cloud, I don't even see a execute action in the inspect logs directed at HA. This seems to indicate an issue with the app.js where it isn't finding a match to teh device. but at times it will as well. No errors either in there.

I've noticed that some entities work with local and some work wit cloud only. I've checked the data in the inspect and I don't really see a difference between the two that would make it not work or work.

two examples:

light.living_room_overhead_light_2 <= always works light.front_office_light <= Never works

If I ask google to force local, the front_office_light will be unavailable. The living_room_overhead_2 works just fine.

[pdobrien3]

interesting. i never get past

but i have only really been trying two switches.

[calisro]

You can expose input_booleans or other devices if you want to test wider in your google_assistant.yaml

You're also on 2022.6.0 (you really should upgrade).

[pdobrien3]

its an old picture. i have been testing since the 2022.6 beta. i have several different types of entities exposed to google but never thought to try a different device. i will try later tonight and report back. good to hear someone else is having issues outside of me. thank you.

[pdobrien3]

I ran through pretty much every device type I have. Nothing works local. Everything works in the cloud

[balloob]

@pdobrien3 the screenshot you're pasting is correct. It has identified your device, selected it as a proxy and asked for reachable devices. You can expand the reachable devices response to see what entities Google will control locally. Note that each device has it's own local SDK running and need to be able to detect Home Assistant.

[balloob]

Can both of you check the execute commands that come in to verify that all devices have a customData set that makes sense ?

[pdobrien3]

99% of my devices exposed to google are zwave devices or input_booleans. ouside of that, a couple wifi thermostats and a chromecast. if you tell me how to check the execute command i will be happy to check .

[pdobrien3]

if the execute commands show in the same place i am getting my screen shots, under chrome://inspect/#devices, i never see anything but the screen shot in posted here https://github.com/NabuCasa/home-assistant-google-assistant-local-sdk/issues/20#issuecomment-1149996576

[balloob]

Yes, the EXECUTE should be logged at the same place as IDENTIFY, PROXY_SELECTED and REACHABLE_DEVICES.

In custom data:

• uuid is used to check if the linked HA instance with Google is the same as the local discovered one. This works as you get a PROXY_SELECTED
• httpPort and webhookId are used to reach the Google Assistant integration in HA.
• All other fields are deprecated and will be removed in the future (and hence why Local SDK 2.1.5+ is required)

[pdobrien3]

can i assume the custom data is correct if everything works using cloud or should i verify the uuid and webhook id?

[balloob]

You don't need to verify the uuid and webhook ID, since you see a REACHABLE_DEVICES, they are correct.

[pdobrien3]

the fact that very few people are having issues is not lost on me. i havent abandoned the idea that something is not set correctly on my network. is there something that changed network wise between version 1.1.0 and 2.1.5?

[balloob]

The fact that Google is asking REACHABLE_DEVICES means it is able to reach your installation.

If EXECUTE targeting any of those devices is not sent locally, it means Google doesn't think those devices are locally reachable 🤷‍♂️

I don't think that the problem is with this SDK.

[calisro]

Can both of you check the execute commands that come in to verify that all devices have a customData set that makes sense ?

All of mine have proper looking customData attributes.

The fact that Google is asking REACHABLE_DEVICES means it is able to reach your installation.

I know mine is slightly different. I see all of these in 'reachable devices'. What I don't see is specific lights/switches reaching back out to HA to execute. Some are, and some aren't. My 'front office light' will simply reach out to cloud. My 'living room overhead light', I will see the execute and it'll work.

This only change after upgrade to 2022.6.x. I barely saw partner_cloud entries prior to then. Most were 'local'. To be fair though, after I got it working, I didn't check too much especiially for specific lights. A good portion of my 66 entities exposed to google work correctly through HA local. THere are odd ones that don't for no rhyme/reason that I can tell.

[balloob]

So that's a QUERY. Queries are used to get the current state of things. They can fire if the screen needs to show the state.

That log is btw not from 2.1.5.

Do either of you have:

• multiple Home Assistant installations
• Share your Google Home with other Google users

[pdobrien3]

i see queries all the time in the google cloud platform logs. updating states. errr....but that is using the cloud i guess. i do not have multiple home assistant installations or share my google minis/homes with other users.

[pdobrien3]

If EXECUTE targeting any of those devices is not sent locally, it means Google doesn't think those devices are locally reachable 🤷‍♂️

so, just to make sure, the only way i can control my devices is to tell google to force cloud. if i just reboot the minis and ask it something it fails, it will not revert to cloud when local fails.

[balloob]

@pdobrien3 what device are you using? If it's a mini, what generation? And you claim it works when you roll back to an older version?

[pdobrien3]

its a mini, prob gen 1 (not sure how to tell and no info under the device in google home) and when i roll back to app.js version 1.1.0 it works perfectly.

and when i asked it, it said "i am version cool.0" 🤣

[calisro]

its a mini, prob gen 1 (not sure how to tell and no info under the device in google home) and when i roll back to app.js version 1.1.0 it works perfectly.

and when i asked it, it said "i am version cool.0" rofl

I believe the Gen 2s have a hole in the underside to wall mount. Gen2s also have a barrel type power cord instead of a usb mini plug. I have both gens. Gen 1s wont do local audio processing as they lack the chip. Gen 2s can. They both can do 'local fullfillment' but in reality, the gen 1s ship the voice processing to the cloud to process. I can force local both sets and force cloud and they will all work but they work differently behind the scenes.

[balloob]

We never published a v1.1.0. We went straight from 1.0.0 to 2.0.0 (commit)

[pdobrien3]

1.1.0 is what shows up when you do the device inspect 🤷🏼. it was one of the first versions. from memory, there was the very first version and then a second one. honestly, i didnt pay attention to versions. the one i have, i got from someone off of discord ( I couldnt find my original download), after having problems with the 2.1.5 version, and it works.

[calisro]

I have both a 1.0.0 and a 1.1.0 and I didn't get one from Discord. I remember specifically have a version and then downloading a new one when going back and forth on this when it was initially released. I have all 3 versions in my downloads folder on my laptop. That was the one I had been running for a very long time and working. I know the because I added a 'v2' marker as a comment in the header of the file so I can make sure it was loaded when inspecting and looking at the js file without searching.

@dell-xps:~/Downloads$ diff app.js app-v2.js
3c3
< /*rc
---
> /*v2
131c131
<         return forwardRequest(hassCustomData, 
---
>         return forwardRequest(hassCustomData,
158c158,171
< const app = new App("1.0.0");
---
> const queryHandler = async (request) => {
>     console.log("QUERY intent:", request);
>     const device = request.inputs[0].payload.devices[0];
>     try {
>         return await forwardRequest(device.customData, device.id, request);
>     } catch (err) {
>         if (err instanceof UnknownInstance) {
>             err.throwHandlerError();
>         }
>         throw err;
>     }
> };
> 
> const app = new App("1.1.0");
163,170c176
<     // Undocumented in TypeScript
<     // Suggested by Googler, seems to work :shrug:
<     // https://github.com/actions-on-google/smart-home-local/issues/1#issuecomment-515706997
<     // @ts-ignore
<     .onProxySelected((req) => {
<     console.log("ProxySelected", req);
<     return createResponse(req, {});
< })
---
>     .onQuery(queryHandler)
178,179d183
<     .onQuery((req) => console.log("Query", req))
<     // @ts-ignore
187,188c191,192
<     console.log("Ready!");
< })
---
>         console.log("Ready!");
>     })

[calisro]

I found where it came from:

https://github.com/home-assistant/home-assistant.io/pull/21491

[balloob]

Alright, so I don't know what is going on, but I am able to reproduce this now. Maybe 100x restarting my hub did the trick. And I can confirm that 1.1.0 fixes it 😭

[calisro]

Alright, so I don't know what is going on, but I am able to reproduce this now. Maybe 100x restarting my hub did the trick. And I can confirm that 1.1.0 fixes it sob

That made me laugh. That was literally me yesterday. Turn the office light on. Turn the office light off. turn............... LMAO

[balloob]

@pdobrien3 can you test something for me: take 2.1.5 and remove the line

.onProxySelected((request) => forwardRequest(Intents.PROXY_SELECTED, request, request.inputs[0].payload.device.id, { supportedHAVersion: [2022, 3] }))

Then see if that works.

[balloob]

No, that's not it. I had broken the version string so commands weren't being processed. I actually am not able to reproduce this locally.

[balloob]

It is it. I can consistently fix a broken setup by removing that line.

[balloob]

Published 2.1.6 with proxy selected handling commented out.

[pdobrien3]

@pdobrien3 can you test something for me: take 2.1.5 and remove the line

.onProxySelected((request) => forwardRequest(Intents.PROXY_SELECTED, request, request.inputs[0].payload.device.id, { supportedHAVersion: [2022, 3] }))

Then see if that works.

do you still need me to test or just try 2.1.6? Sorry, was working in the yard all morning.

[balloob]

Just try 2.1.6, it has the fix.

[pdobrien3]

Thank you Paulus. That seemed to fix things on my end. Appreciate you hanging in there with me. Hopefully it fixes Calisro's issues also.

[calisro]

No. Mine isn't fixed. But frankly I am unclear if its on HA side or Google's side. Its really a wierd thing.