[xcode10] Siri shortcuts - custom responses not appearing

[bradumbaugh]

Steps to Reproduce

1. Download Apple's SoupChef sample.
2. Run this sample; create a soup order; assign a Siri shortcut to it; invoke the Siri shortcut; to confirm the order, say "Yes" rather than tapping on the button in the UI.
3. Note how after the soup order completes, Siri indicates (audibly) that it will be ready to pick up in ten minutes.
4. Download the Xamarin SoupChef port
5. Run the same test on the Xamarin port, but note that Siri does not read the custom response text
6. It appears that the correct completion handler is getting called in OrderSoupIntentHandler in the HandleOrderSoup method, but it's not having the anticipated effect. To see this, debug the SoupChefIntents project and set a breakpoint on the last line of the HandleOrderSoup method, where the completion handler is called.
   • Is there an issue with the way the completion handler is bound? (See the OrderSoupIntentBinding project in the same solution)
   • Note that the OrderSoupIntentResponse.SuccessIntentResponseWithSoup method doesn't seem to be the problem, as I attempted to just manually create the OrderSoupIntentResponse object and pass it to the completion handler. Same result.

Expected Behavior

Siri should indicate audibly (and in text in the UI) that the order will be ready in 10 minutes, as it does in the Swift-based example.

Actual Behavior

Siri completes the order, but does not indicate that it will be ready in ten minutes.

Environment

• Xcode 10 beta 5

[bradumbaugh]

One further note on this: it seems like there can sometimes be a lag between requesting that Siri execute a custom intent and the actual execution/sometimes a failure. This also happens some with the Swift-based sample, but it's worth being aware of.

[bradumbaugh]

This sample is now in ios-samples: https://github.com/xamarin/ios-samples/tree/master/ios12/SoupChef

[VincentDondain]

The beginning of my investigation leads me to believe there's a mismatch between the types we send to OrderSoupIntentResponse and the types the .intentdefinition file expect. Note: I noticed you modified the .intentdefinition file.

I think the completion handler is bound correctly (my 👀 ) and works (because Siri tells you there's a success and finishes the intent).

With the native sample, if for instance you do not pass both the soup and waitTime to OrderSoupIntentResponse for success (which expects both values) you then have the same behavior we have in the Xamarin sample. Aka, it tells you it worked without the verbal detailed confirmation.

In addition I cannot use the FailureSoupUnavailable (that one fails completely).

I'm not quite sure how to debug properly the intent (the debugger works but that's not what I want). I looked at the device console output but it's very verbose and I couldn't find what I wanted yet.

To be continued...

[bradumbaugh]

@VincentDondain – I didn't modify the .intentdefinition file, but it looks like Apple has updated the sample, so I'll need to update the port as well to go along with the changes.

[VincentDondain]

So I don't think it's a type mismatch anymore because a failure or success message with a simple message (no special values) in the .intentdefinition doesn't work either. Somehow Siri is simply not able to pick/read the custom response messages and I still don't know how to properly debug that because it happens after the completion of the IntentHandling. I'd expect some error to be printed in the device logs but so far I couldn't see anything.

[VincentDondain]

Also I know Siri can "read" the .intentdefinition file because changing the category in that file has an effect on the Siri response words.

In addition the error codes seem to work because failure and success are default error codes and Siri will say that an error happened or that it was successful.

[VincentDondain]

Maybe it's the INIntentResponseCodeFormatStringID that is used by Siri to show the INIntentResponseCodeFormatString. However I do not see anything "using" that id...

[VincentDondain]

Braindump / recap:

1. We know Siri / the Intent Extension can read the .intentdefinition file because if we change the Category from order to like download. All the Siri responses will move away from the word "Order" and instead use "Download". Note: this is ruling out any resource issue.
2. We know that the OrderSoupIntentHandling and in particular HandleOrderSoup work because we can debug it and calling (or not calling) completion will complete (or not) the execution of the intent. Note: this is proving we have a somewhat correct binding project.
3. We can trigger different Siri responses based on either OrderSoupIntentResponseCode.Failure or OrderSoupIntentResponseCode.Success. Siri is either telling you everything works and it completes the order or that there was an error and offers to open the app. So we know that the enum values have an effect and that it's probably mapped to the .intentdefiniton where we have INIntentResponseCodeName = failure and INIntentResponseCodeSuccess = false. That mapping of the enum values to the .intentdefinition file's enum names here is still confusing to me.
4. Now for a success or failure of the INIntentResponse, the INIntentResponseCodeFormatString (audible response) should be used by Siri and it is not. It doesn't work without any special parameters given to INIntentResponseCodeFormatString and by passing new OrderSoupIntentResponse (OrderSoupIntentResponseCode.Success, null) as a response to the completion handler completion (new OrderSoupIntentResponse (OrderSoupIntentResponseCode.Success, null));. Note: this is ruling out my previous theory of custom parameter type mismatch between the .intentdefinition file and the binding project.

[VincentDondain]

Provisioning for device instructions: https://github.com/xamarin/ios-samples/tree/master/ios12/SoupChef#running-on-device

I actually did not try on simulator but it should work and be much easier to setup (no setup) (:

[Odrakir]

Hi, we are trying to incorporate Siri Shortcuts to our Xamarin app, and everything works fine until we try to get a custom voice response from Siri.

Any estimations on when will this work? or any workarounds for this problem?

Thanks,

[VincentDondain]

Hi @Odrakir, not yet unfortunately, we are still working on this. I will update this issue whenever we have some progress.

[VincentDondain]

Twitt from Manuel: https://twitter.com/mandel_macaque/status/1034204628705923072. That might indicate some missing settings on our test case project.

I couldn't yet understand that Xcode auto importing classes magic.

[Odrakir]

Not sure if this helps, but if I try this from the Apple Watch, Siri does say the custom responses just fine.

[Odrakir]

Hi, any news at all about this? We are integrating more Siri features in our app, but the lack of custom responses is a major flaw for us.

If there's anything we can do to help debug this problem, please let us know.

[Odrakir]

Oh well, in case anyone is still interested, I managed to fix this. Apparently it's not directly related to Xamarin, but to localisation. I added localisation for the intent definition file and made sure I imported all localised versions in the shared project.

Reference: https://stackoverflow.com/questions/52618351/siri-shortcuts-response-templates-not-working

[chamons]

@Odrakir Unfortunately, we don't have any news to share right now.

Until we add support, you could look at building a minimal extension in Obj-C and copying into your Xamarin.iOS final bundle.

This has been done in the past when people have run into issues:

https://bugzilla.xamarin.com/show_bug.cgi?id=43985#c12 https://stackoverflow.com/a/43647608/36782

That second link has detailed instructions that may be useful.

[Odrakir]

That's interesting. Thanks.

But as I said in my second message it's been fixed, and it was related to localization and not Xamarin, apparently.

[VincentDondain]

Oh wow! Super cool thanks for sharing the fix, I'll try to update our SoupChef sample later, glad you're unblocked (:

[Vignesh-Ram]

I am using SIRI Custom Intent for voice response. It is not working at all.
Referred to Souf Chef: https://developer.xamarin.com/samples/monotouch/ios12/SoupChef/

NSUseractivity is working. But HandleIntent is not calling