Recent Firmware Upgrade Broke Local Fulfillment

[pdobrien3]

During the early morning hours of 11/22, I believe all my google minis upgraded their firmware to 1.56.324896 (although I have been unable to confirm this on the minis other than the current firmware version). A google search produced this:

Get started with Matter - Google Home Developershttps://developers.home.google.com › matter › get-started Jul 14, 2022 — The Nest Hub (2nd gen) must be on Cast firmware version 1.56.324896 or higher. A mobile device (test phone) that meets the minimum ...

https://developers.home.google.com/matter/get-started

Local fulfillment stop working on 11/22/2022

{ insertId: jsonPayload: { appVersion: "2.1.6" errorCode: "COMMAND_FAILED" intent: "REACHABLE_DEVICES" message: "{"requestId":"xxxxxxxxxxxxxxxxxxxxxxxxx","payload":{"errorCode":"COMMAND_FAILED","debugString":"HTTP response body not set."}}" requestId: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" severity: "ERROR" timestamp: "2022-11-22T10:48:16.579Z" }

[balloob]

Check for errors in your HA log, otherwise enable debug logging for google_assistant to see the response sent.

[pdobrien3]

with this in configuration.yaml:

logger:
  default: info
  logs:
    homeassitant.components.google_assistant: debug
    google_assistant: debug

and a request of the mini to turn on the lights, i get, sorry, it looks like the office light isnt available right now and nothing in the HAss logs.

[pdobrien3]

I was hoping the beta would fix it. I know it was a long shot but...... I also removed my local fulfillment settings and controlling devices still doesn't work. Is this something that is probably just going to have to wait for the rest of the world to get on this firmware version or is there something else I can try?

[balloob]

Well, like the last time, it seems like your setup is special 🙃

[pdobrien3]

hmmm.....except this time, someone else has experienced the same issue according to the thumbs up on the original post. I am more than happy to admit that i am a novice when it comes to this stuff but having the googles break the same time that Matter notification comes out 🤷🏼

[Tugzrida]

I seem to have the same issue. Local fulfilment was working fine until a few weeks ago.

I've spent quite a while looking into it today and I can't find much. The cloud SYNC and local IDENTIFY and REACHABLE_DEVICES intent requests and responses all look correct compared to REQUEST-RESPONSE.md and the Google documentation.

I can see a local REACHABLE_DEVICES request and response every minute as expected, however after "Hey Google, force local", every command is met with "Sorry, it looks like the XXX isn't available right now". Throughout this process Wireshark shows no packets between the Home Mini and HA. No logs show up in Google Cloud or the local fulfilment console via Chrome. From this, it seems that the Home Mini decides that the devices aren't available for local fulfilment in advance, rather than trying to connect to HA and failing.

After resetting the behaviour with "Hey Google, force default", everything works again, albeit via cloud fulfilment.

It unfortunately looks like an issue with the Home Mini firmware itself and not HA's implementation of the SDK.

[johnstav]

i'm also having this issue. identify and reachable_devices is replying fine. google logs show partner_cloud execution. force local gives the same isn't available error.

[Victorsueca]

[...] however after "Hey Google, force local", every command is met with "Sorry, it looks like the XXX isn't available right now". Throughout this process Wireshark shows no packets between the Home Mini and HA. No logs show up in Google Cloud or the local fulfilment console via Chrome. From this, it seems that the Home Mini decides that the devices aren't available for local fulfilment in advance, rather than trying to connect to HA and failing.

After resetting the behaviour with "Hey Google, force default", everything works again, albeit via cloud fulfilment.

I'm experiencing this exact behaviour here too. As a side note, I am using apache as a reverse proxy for connections from the outside, and my first thought was that could be affecting it somehow, but it used to be working before, and it stopped working for no apparent reason.

[cdemi]

I have this problem as well and I have also moved to the Preview Program. If you need any logs or some specific information I can provide

[shiznat]

Just chiming in to confirm I am having the same problem. Local fulfilment was working until the 8th of December (I can see via Google Cloud console that local fulfilment requests dropped off a cliff at exactly that time, presumably when Google rolled out a firmware update to my devices).

Making Google Assistant use 'force local' gives me the same error as others - "Sorry, it looks like the XXX isn't available right now" and switch back to 'force default' and everything works again, albeit much slower!

[balloob]

We're now seeing it at Nabu Casa too, but not for everyone. We're talking to Google and they are looking into it.

[wjoshraymond]

I am seeing this same behavior as well. Started about 2 weeks ago, where everything prior was working perfectly. The error message I'm seeing from chrome://inspect is HTTP response body not set. Force local does not work, and I tried reconnecting HA with Google, which succeeded, but yielded no change in local fulfillment behavior.

Please let me know if there's any additional testing or logging that would be helpful.

[nevir]

I'm new to this project, so I'm not sure what the behavior was prior to 2.1.6, but at least for my setup, I'm seeing a command being generated with port: 443, protocol: 'HTTP' (and the port is coming from HA's customData …but I don't yet understand how that gets passed to Google Assistant).

Assuming that the GA libraries are connecting to <my-ha-host-ip>:443 and trying to talk HTTP to them, that just isn't going to be able to work with my setup (NGINX proxy manager is listening to 443 for HTTPS, not HTTP)

[Tugzrida]

@nevir I have the same issue open at https://github.com/home-assistant/core/issues/80063

The short version is that Google Assistant can only use HTTP for local fulfilment, but the port that's provided to Google Assistant is taken from the configured "Internal URL" in HA, so if you've set the Internal URL to HTTPS, local fulfilment won't work, even if HA is accessible over HTTP on some other port. Ideally the port would be configurable in HA, but it currently isn't. My workaround at the moment is to change the reference to deviceData.httpPort in the Google Assistant JS file to 8123, the port where HA is listening on HTTP in my setup.

[pdobrien3]

My internal url is set to a local IP address with port 8123 and my external url is set to use https with my domain and nginx/swag and local fulfillment is broken for me. I also tried changing deviceData.httpPort to 8123 without any change.

[Tugzrida]

@pdobrien3 The issue in this thread is on Google Assistant's side, where local fulfilment seems to be broken for many people at the moment.

The issue with ports and HTTPS is seperate and being tracked at https://github.com/home-assistant/core/issues/80063. If your internal URL is set to port 8123, then the issue with ports will not affect you.

[GodlyOverlord]

So is this just a "sit back and wait" type issue for majority of us then until someone either figures out whats happening/if it can be fixed/worked around on HA's side or google them selves sorts it out?

[balloob]

We need to wait until Google sorts it out. We didn't change anything.

[GodlyOverlord]

We need to wait until Google sorts it out. We didn't change anything.

They didn't change any requirements though, its just flat out broken on their end? (ie the app.js doesn't need anything modified to suit some new requirement they pushed out?)

We'll just have to make do with cloud fulfillment for now then

[shiznat]

I am seeing local fulfilment working again without making any changes on my end. How about anyone else?

[wjoshraymond]

Just tried force local after reconnecting accounts and no joy on my end.

[pdobrien3]

I am seeing local fulfilment working again without making any changes on my end. How about anyone else?

Curious if the firmware version on your google device changed. Firmware seems to roll out at different times for these devices. My minis are still on 1.56.324896.

This https://support.google.com/googlenest/answer/7365257?hl=en#zippy=%2Ccurrent-production-firmware-version%2Ccurrent-preview-program-firmware-version seems to indicate 1.56.324896 is current for both production and the preview program but that doc was update on 01/11.

[wjoshraymond]

Update: Randomly started working tonight with no changes on my end. I'll look into firmware versions this weekend and report back.

[benrafshoon]

Confirmed that local fulfillment is now working for me again!

I have two Google Assistant devices on my network, a Google Home Mini and a 1st gen Nest Hub. Based on chrome://inspect logs, local fulfillment is going through the Nest Hub even when I use a voice command with the Google Home Mini. I'm on the following firmware versions:

• Google Home Mini: 1.56.324896
• Nest Hub 1st gen: 9.20221010.3.176

The firmware version on my 1st gen Nest Hub is marked as the "Preview Program" version on https://support.google.com/googlenest/answer/7365257, while the firmware version on the Google Home Mini is the "Production" version (or rather, the current firmware version for the Google Home Mini is the same for "Preview Program" and "Production").

I wonder if the people who are now seeing local fulfillment working are those that have devices updated to the "2022-10-10" firmware version (that's 1st gen Nest Hubs and Nest Hub Maxes in the preview program per https://support.google.com/googlenest/answer/7365257).

[dyay108]

Still broken for me, both 2nd gen mini (nest) and 1st gen mini on latest firmware (1.56.324896)

[Victorsueca]

Google Nest Mini @ version 324896 here, it started working with force local 2 or 3 days ago, stoped working again this afternoon.

[balloob]

Rollout that fixes the current issue is happening. It takes time for your device to get it, you cannot skip ahead in the queue.

[sagarspatil]

Rollout that fixes the current issue is happening. It takes time for your device to get it, you cannot skip ahead in the queue.

Local fulfillment is still not working for me. When I force local, it says 'It looks like the device is not available' but when I 'force default' it works. Is there anyway to get local to work again?

[balloob]

If you are experiencing issues, please open a new issue. Include logs, versions used etc. The cause of this issue has been solved.