[Xamarin.UITest] iOS 14 springboard support for sh.calaba.DeviceAgent.xctrunner

[ccxla]

Describe the solution you'd like Before iOS 14, the old DeviceAgent-Runner was able to dismiss springboard alerts such as iOS prompting for camera permissions for an app. As of Xamarin.UITest 3.0.11 (cf. https://github.com/microsoft/appcenter/issues/2050) with sh.calaba.DeviceAgent.xctrunner, this does not work anymore.

System.AggregateException: One or more errors occurred. (DeviceAgent springboard-dismiss-alerts failed: A SpringBoard alert is blocking test execution and it cannot be dismissed.) ---> Xamarin.UITest.XDB.Exceptions.DeviceAgentException: DeviceAgent springboard-dismiss-alerts failed: A SpringBoard alert is blocking test execution and it cannot be dismissed.
  at Xamarin.UITest.XDB.Services.iOSDeviceAgentService+DeviceAgentResult.Validate (System.String action) [0x00028] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.XDB.Services.iOSDeviceAgentService.RequestAsync[T] (System.String deviceAddress, System.String route, System.Object data, System.String action) [0x00102] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.XDB.Services.iOSDeviceAgentService.DismissSpringboardAlertsAsync (System.String deviceAddress) [0x00080] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00013] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2029 
  at System.Threading.Tasks.Task.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00043] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2759 
  at System.Threading.Tasks.Task.Wait () [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2625 
  at Xamarin.UITest.iOS.iOSGestures.InternalQuery[T] (Xamarin.UITest.Queries.AppQuery query, System.Object[] args, System.String methodName) [0x00019] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.iOS.iOSGestures.Query (Xamarin.UITest.Queries.AppQuery query) [0x0000d] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.iOS.iOSApp+<>c__DisplayClass14_0.<Query>b__0 () [0x00017] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.Utils.ErrorReporting.With[T] (System.Func`1[TResult] func, System.Object[] args, System.String memberName) [0x0000e] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
---> (Inner Exception #0) Xamarin.UITest.XDB.Exceptions.DeviceAgentException: DeviceAgent springboard-dismiss-alerts failed: A SpringBoard alert is blocking test execution and it cannot be dismissed.
  at Xamarin.UITest.XDB.Services.iOSDeviceAgentService+DeviceAgentResult.Validate (System.String action) [0x00028] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.XDB.Services.iOSDeviceAgentService.RequestAsync[T] (System.String deviceAddress, System.String route, System.Object data, System.String action) [0x00102] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 
  at Xamarin.UITest.XDB.Services.iOSDeviceAgentService.DismissSpringboardAlertsAsync (System.String deviceAddress) [0x00080] in <e0d157665baf4a5bbaccde702d8fe0bf>:0 <---

Describe alternatives you've considered none

Additional context This issue is similar to https://github.com/microsoft/appcenter/issues/1912.

[IlyaBausovAkvelon]

Good Day, Could you please specify what iOS version do you use? 14.0 or 14.2?

[Banley]

I am seeing the same issue. Our devices are on iOS 14.1, using DeviceAgent sh.calaba.DeviceAgent.xctrunner, Xamarin.UITest 3.0.11.

[ccxla]

I am on iOS 14.2.

[IlyaBausovAkvelon]

@Banley @ccxla On what locale are you running tests? We have some issues with alerts in DeviceAgent and probably fixed them, but to be sure we need to know what locales are affected by this.

[ccxla]

@IlyaBausovAkvelon I tried with English (UK) and English (US), both yield the same error.

[Banley]

Running tests in US. Still getting the error today.

[ccxla]

and probably fixed them

@IlyaBausovAkvelon Any chance to get the pre-release that contains the fixes you mentioned? On a side note, once again I would like to suggest open sourcing this component so that the community can iterate on these fixes more quickly.

[IlyaBausovAkvelon]

Hello, What alerts are exactly blocking you? Could you please provide the screenshot of alert that aren't being handled? We are adding exact handlers for each alert so we should now what alert are creating the issue.

[IlyaBausovAkvelon]

Is it camera alert or Network discovery permission alert?

[IlyaBausovAkvelon]

I see that camera and network discovery alerts are already in 3.0.11.

[Banley]

The alert I see right now locally is this: ""my-app-name" Would Like to Send You Notifications. Notifications may include alerts, sounds, and icon badges. These can be configured in Settings." I think the alert will come up with any fresh app install. Also, some of my issues are possibly due to user error/updates. I got my local mac working with xcode 12.2 and physical ipad on iPadOS 14.2. Updated the remote devices accordingly and hopefully that fixes everything for me, but will need to manually dismiss any Notification alerts for fresh app installs. This will be unacceptable for most users here.

[ccxla]

I see that camera and network discovery alerts are already in 3.0.11.

@IlyaBausovAkvelon Unfortunately I cannot confirm this (cf. https://github.com/microsoft/appcenter/issues/2060#issue-755341434). The alert says "[App]" Would Like to Access the Camera, just like in the example below.

[IlyaBausovAkvelon]

Could you try to add this line (if you have not done it already) to your tests in the place where alert is expected to appear: app.DismissSpringboardAlerts();

[ccxla]

@IlyaBausovAkvelon Thanks, that does actually work.

However I would consider it a workaround since it requires app to be iOSApp, when in older versions of Xamarin.UITest, this action was performed automagially whenever there was a blocking springboard alert.

[IlyaBausovAkvelon]

I am glad to hear that this workaround has worked for you. You are right, this is a temporary workaround for not blocking you. I am still in the process of investigating what is wrong with automatic dismissing of alerts. I will let you know when the issue will be resolved.

[ccxla]

I will let you know when the issue will be resolved.

@IlyaBausovAkvelon Any news on this? Should I make a new ticket to track the automatic dismissal issue?

[mlancione]

I'm experiencing the same issue. The app.DismissSpringboardAlerts() workaround doesn't dismiss the popup for me. The popup says: '"XXXX" Would Like to Access the Camera'.

[ChristopherStephan]

We have the same error with the Local Network Usage Permission dialog. It looks like the Device Agent implementation is missing an entry for this kind of dialog.

https://github.com/calabash/DeviceAgent.iOS/blob/b6ac427308ed5b4a58ddc7011d04ccc13754f707/Server/Utilities/SpringBoardAlerts.m

(App as Xamarin.UITest.iOS.iOSApp).DismissSpringboardAlerts(); worked for me as a workaround.

[edgiardina]

My app gets this alert as the first thing that happens in app (network permission dialog like ChristoperStephan indicated) , on my MSAL screen. However I can't seem to DismissSpringboardAlerts() because of a MediaType exception.

Test Logs here https://gist.github.com/edgiardina/042a236eb60e251cdeee18fe2673fad1

Xamarin UITest 3.2.7 Xamarin Testcloud Agent 0.23.1 Nunit 3 test adaper v 4.2.1 Nunit 3.13.3

running unit tests on my local simulator I do not see any issues. It's only testing in AppCenter that I get that dialog, and only on iOS 14 and up. (iOS 13.7 I don't see the issues in AppCenter)