Google Assistant is recommending the device before executing

[ole1986]

While I have properly configured the actions.json for a custom device action using a webhook (instead of a "staticFulfillment") I noticed that google assistant always ask me to "use" the appropriated device first, before the webhook is being called. Which I always have to confirm with "yes"

I am wondering why this is happening?

Does it have something to do with the below"availabilty" definition?

"availability": {
    "deviceClasses": [{
            "assistantSdkDevice": {}
        }
    ]
},

I haven't implemented such defintion as I do not know for what this stands for. Anyone who can explain me what this is?

[Fleker]

Custom device actions aren't meant to have a webhook for fulfillment.

[ole1986]

@Fleker thank you for the answer. But honestly I do not really understand how it will affect the implementation?

As I can tell it is working so far, only I have to answer with "yes" until action is triggered. Will this be always the case and do i have to implement it as staticFulfillment to avoid Google Assistant asking me such question?

[Fleker]

Using a webhook for fulfillment was never intended, so I'm not really sure what would happen in that case. It's indeterminate behavior.

[ole1986]

How and why would this be handled differently compared to a "regular device" and what does this exactly mean for my custom device?

Does it mean when using webhooks I always have to start the conversation with "Hey Google, talk to [...]" followed by "and do me a coffee"?

In addition to this, I noted whenever I say that, google assistant respond to me with "Okay, Starten wir die Testversion von [...]" followed by the custom (matching) intent.

Is this something because I haven't released it yet in the google actions console?

I am sorry to bother you but I actually could not really find any good parts in the documentation about this topic.

What I have found was: https://stackoverflow.com/a/47835391/2106997

[Fleker]

There are several different types of Actions with specific ways that they work. While similar in some ways, there are distinct models for how they are meant to be developed and used.

For an Assistant SDK device, with custom device actions, it is meant to have a static fulfillment that sends a response, and a callback in your code. I'm surprised that a webhook works at all. Since the systems are similar, I guess that does make it possible. It's pretty much like you're using Actions SDK instead of custom device actions. With this, you should be able to give direct commands like "move forward" that make sense for your device.

Conversational actions are designed to be widely accessible on devices. To make them available across surfaces, they have a webhook which does all the processing and sends a response which can be rendered differently on different surfaces. Actions SDK is one way to do that, and both this and device actions use an actions.json file to provide all of the metadata. Conversations are started with "Talk to XXX" to make it explicit that you're invoking an action and not doing something else.

There's also smart home, which is similar but also different.

This is why I'm a bit puzzled. In these two development models, you wouldn't have a custom device action with a webhook nor have a conversational action with static fulfillment. Based on what you did, maybe it works due to some way that the two were developed, but it's unexpected and undocumented.

[ole1986]

Thank you for the detailed answer, really appreciate it Can you please include a short explanation on how Smart Home differs from the other two.

Thank you in advanced

[Fleker]

Smart home is a different model that is based around a webhook primarily (with a local execution path that was just announced). The webhook that you develop doesn't take in user queries directly. Instead, the Assistant handles the natural language processing and abstracts the user's request into a structured request. For example, saying "Preheat the oven to 450" would be turned into a command like "device: oven, temperature: 450".

As the grammars are handled by the Assistant, users can give commands directly (without saying "talk to") and commands are available on any surface (your phone, Google Home, etc.). At the same time, since Google manages the grammars, you don't have the same flexibility over the types of queries you accept, nor can you provide custom text responses.

One additional benefit is that smart home enabled devices do not need the Assistant built into them, unlike Assistant SDK. I can buy any device that has any number of capabilities, but does not need its own microphone. As long as it has a Wi-Fi connection, it will work.

[ole1986]

It is pretty hard for me to make the right choice.

To summarize it (please correct it when I am wrong):

Assistant SDK:

• actions.json usually contains staticFulfillment as fulfillment
• allowing me to say "Hey Google, do my action" directly (by using the custom intent "com.example.intents.myaction" defined in actions.json)
• allowing me to call device commands being executed locally
• usually requires assistant to be installed (E.g. python sdk)

Actions SDK:

• actions.json usually contains conversations webhook as fulfillments
• does not allow me to say "Hey Google, do my action" for some reason
• does not allow me to first say "Hey Google, talk to my-device" followed by "do my action" (no custom intents defined in actions.json are being triggered afterwards)
• allow me to say "Hey Google, talk to my-device and do my action" for some reason (by using the custom intent "com.example.intents.myaction" defined in actions.json)

Smart Home:

• usually no actions.json required, since it uses the fulfillment webhook for SYNC, QUERY, EXECUTE intents
• does support direct commands (like "Hey Google, close the shutters") dependent on the device types and traits being synced
• for some reason i can still upload an actions.json. Question: can i still use SYNC, QUERY, EXECUTE when an actions.json is available?
• no custom responses, like SimpleResponse, BasicCard, ... provided by Actions SDK

Is there any good explanation of why I can upload an action.json (usually used by Actions SDK / Assistant SDK) into a smart home configuration? And will both be able to work together?

Because, this is exactly what I need. Being able to give direct commands without loosing the ability of Actions SDK using webhooks. Of course dedicated to the end user

Sorry for the long text. I am convinced this will make it more clear to me

[ole1986]

I noted after uploading the actions.json into my existing smarthome Google Actions project, both the smart home and actionssdk intents are being executed.

What I did

• Removed all actions from the project (by remove the language)
• Added a new language (E.g "german")
• Completed the fulfillment url for Smarthome webhooks and pressed "Save"
• Uploaded the actions.json using gactions

Now the actions are available but I cannot access the Smarthome webhook url anymore from the "Actions" menu. Still, both are being executed.

Which I hope is a feature and not a bug :-)

[Fleker]

Your summaries are mostly correct.

You should be able to first say "OKG, talk to my-device" followed by "do my action" is a follow-up query once you've entered the conversation.

For smart home Actions, an action package was the original way to configure actions, before everything moved into the console. You should not expect to keep doing this.

For any given project, you should not expect that Actions SDK, Assistant SDK, and smart home work together. As they're all different, there's not a reason why you'd want more than one tied to the same project.

[ole1986]

@Fleker thank you very much for your attention.

As you were already mentioning they are same, but different, still same (meme alert!) Unless smart home does not provide these options covered by actions sdk I need to handle it in both.

I already requested an amendment on devices type "action.devices.types.SHUTTER" to fulfill at least a minimum of it.

Thank you