[Bug]: IOS Real Device Tests started failing

agray commented 1 month ago

Description

Real Device tests started failing after upgrade to Appium dotnet client 5.1.0 and Selenium 4.23.0 on Saturday 10/08 on SauceLabs.

Environment

.NET client build version or git revision if you use some snapshot: 5.1.0 and Selenium 4.23.0
Mobile platform/version under test: IOS devices
Real device or emulator/simulator: Real Devices

Details

SauceLabs tests for iPhone 15 Pro Max Real Device started failing on Saturday 10th August.

Please provide more details, if necessary. Error returned from SauceLabs: "Status Code: 500 - Error while starting appium session: An unknown server-side error occurred while processing the command. Original error: No webview has been detected after 13926ms."

Dor-bl commented 1 month ago

@agray, have you checked the https://github.com/appium/dotnet-client#compatibility-matrix?

agray commented 1 month ago

I am using Selenium 4.23.0

Dor-bl commented 1 month ago

And dotnet client version 5.0.0, works for you?

KazuCocoa commented 1 month ago

Could you attache the full Appium server log?

No webview has been detected after 13926ms thinks me that it was client independent behavior. It would be helpful to attach working Appium server log as well.

agray commented 1 month ago

Rolled back to Appium 5.0.0 and Selenium 4.22.0 and even less works. (I had originally skipped Selenium 4.22.0.)

https://github.com/Sauceforge/Saucery/commit/32ef411ac19b2c68a5215240caaf76d3c864ca31

NUnit Virtual: Tests launch (except IOS Virtual) XUnit Virtual: Tests don't launch NUnit Real Devices: Tests don't launch XUnit Real Devices: Tests don't launch

On Appium 5.1.0 and Selenium 4.23.0:

NUnit Virtual: Tests launch XUnit Virtual: Tests launch NUnit Real Devices: Tests launch (only Android works) XUnit Real Devices: Tests launch (only Android works)

agray commented 1 month ago

Could you attache the full Appium server log?

No webview has been detected after 13926ms thinks me that it was client independent behavior. It would be helpful to attach working Appium server log as well.

https://app.saucelabs.com/tests/5988caa82b3d4cbb858bf745f5ca7d24#0

agray commented 1 month ago

It last worked on August 1st: https://app.saucelabs.com/builds/rdc/ccadaa7715f33b228031f806c6f56087

IPhone 15 Pro Max (IOS 17.4.1) https://app.saucelabs.com/tests/13c99c1b6c674651af2f952ec2113e2d

I'll even provide the video from SauceLabs: https://github.com/user-attachments/assets/bae2ea75-58b7-4333-b885-a558926f034d

Dor-bl commented 1 month ago

So what was the last version that worked for you properly?

agray commented 1 month ago

So what was the last version that worked for you properly?

I think it was pre 5.0.0 (or one of the 5 betas) and Selenium 4.21.0

Dor-bl commented 1 month ago

Does other webViews works for you? I have no major changes in mind that can cause this.

agray commented 1 month ago

Does other webViews works for you? I have no major changes in mind that can cause this.

Webviews?

KazuCocoa commented 1 month ago

No available web pages after 10 retries: Could not connect to a valid webapp. Make sure it is debuggable and has at least one active page.

This is appium log from the failed one, 5988ca.... It looks like the appium xcuitest failed to get webviews. I haven't checked logs well yet but it is client independent.

agray commented 1 month ago

It was working on 1st August and I have only updated nuget packages. This code base has worked for many many years.

Dor-bl commented 1 month ago

Does other webViews works for you? I have no major changes in mind that can cause this.

Webviews?

https://developer.apple.com/design/human-interface-guidelines/web-views

KazuCocoa commented 1 month ago

worked one:

{"level":"VERBOSE","message":"[Appium] Requiring driver at /root/appium/appium2-202408010003/node_modules/appium-xcuitest-driver/build/index.js","timestamp":"2024-08-01 10:18:19.926"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Appium v2.11.2 creating new XCUITestDriver (v7.24.6) session","timestamp":"2024-08-01 10:18:19.927"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Checking BaseDriver versions for Appium and XCUITestDriver","timestamp":"2024-08-01 10:18:19.928"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Appium's BaseDriver version is 9.11.2","timestamp":"2024-08-01 10:18:19.928"}
...
ebugger] Notified that an application has been updated","timestamp":"2024-08-01 10:18:22.677"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Current applications available:","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]     Application: \"PID:504\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         id: \"PID:504\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isProxy: false","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         name: \"Safari\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         bundleId: \"com.apple.mobilesafari\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isActive: true","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isAutomationEnabled: true","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         pageArray:","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]           - id: 1","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             title: \"\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             url: \"about:blank\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             isKey: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]     Application: \"PID:209\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         id: \"PID:209\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isProxy: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         name: \"amsengagementd\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         bundleId: \"com.apple.amsengagementd\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isActive: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isAutomationEnabled: \"Unknown\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Finally selecting app PID:504","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Selected app after 10ms","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][XCUITestDriver@4505] Picking webview 'WEBVIEW_504.1' after 292ms","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][XCUITestDriver@4505] Attempting to set context to 'WEBVIEW_504.1' from 'NATIVE_APP'","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Selecting page '1' on app 'PID:504' and forwarding socket setup","timestamp":"2024-08-01 10:18:22.681"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Sending '_rpc_forwardIndicateWebView:' message to app 'PID:504', page '1' (id: 4): 'indic

issue's one:

{"level":"VERBOSE","message":"[AppiumDriver@45ad] Appium v2.11.3 creating new XCUITestDriver (v7.24.14) session","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] Checking BaseDriver versions for Appium and XCUITestDriver","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] Appium's BaseDriver version is 9.11.3","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] XCUITestDriver's BaseDriver version is 9.11.3","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[XCUITestDriver@52b5] ","timestamp":"2024-08-11 00:51:07.671"}
...
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Received response from send (id: 2): '[\"PID:606\",{}]'","timestamp":"2024-08-11 00:51:14.406"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Sending to Web Inspector took 5ms","timestamp":"2024-08-11 00:51:14.407"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Skipping app 'PID:606'. Original error: Empty page dictionary received","timestamp":"2024-08-11 00:51:14.414"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.415"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.533"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.535"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.537"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Current applications available:","timestamp":"2024-08-11 00:51:14.915"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]     Application: \"PID:606\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         id: \"PID:606\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isProxy: false","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         name: \"Safari\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         bundleId: \"com.apple.mobilesafari\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isActive: true","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isAutomationEnabled: true","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         pageArray: undefined","timestamp":"2024-08-11 00:51:14.917"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]     Application: \"PID:398\"","timestamp":"2024-08-11 00:51:14.917"}

Your automation used different version of xcuitest driver. I wondered if recent refactoring in remote debugger affected pageArray selection

cc @mykola-mokhnach (just to share the remote debugger's thing)

Or... https://github.com/appium/appium-xcuitest-driver/blob/master/CHANGELOG.md#7249-2024-08-01 affected the behavior? My recent local test with browserName safari worked though. Need to debug

KazuCocoa commented 1 month ago

Could you test again with XCUITest driver v7.24.6 once? @agray

agray commented 1 month ago

I have never specified the version of XCUITest

See SauceLabs platform configurator documentation https://app.saucelabs.com/platform-configurator

SUT is page hosted by SauceLabs: http://saucelabs.com/test/guinea-pig

mykola-mokhnach commented 1 month ago

@agray Is the issue persistent, e. g. does it happen always or rather randomly? Are you able to reproduce the issue locally?

Could you please provide an example of a failed log with appium:safariLogAllCommunication capability set to true?

mykola-mokhnach commented 1 month ago

Also please review logs attached above - they contain your user name and the access token. I don't think you want these values being shared

agray commented 1 month ago

so please review logs attached above - they contain your user name and the access token. I don't think you want these values being shared

deleted

agray commented 1 month ago

@agray Is the issue persistent, e. g. does it happen always or rather randomly? Are you able to reproduce the issue locally?

Could you please provide an example of a failed log with appium:safariLogAllCommunication capability set to true?

Yes its persistent and I get the same exception locally while trying to initialise the driver on line 93 of SauceryBase.cs trying to instantiate a new IOSDriver. This is an integration test so it has to talk to SauceLabs.

mykola-mokhnach commented 1 month ago

Does it also happen on a single device or multiple devices? Have you tried to reboot the device if it is always the same and run the test again?

Eventually, there is also a possibility on Sauce to explicitly provide the Appium snapshot version to run on (https://docs.saucelabs.com/mobile-apps/automated-testing/appium/appium-versions/). You may try to run with an older version to verify the assumption whether the issue has to do with Appium-specific changes

agray commented 1 month ago

Does it also happen on a single device or multiple devices? Have you tried to reboot the device if it is always the same and run the test again?

The "device" is the one requested via DesiredOptions on SauceLabs. SauceLabs spins up a new instance for EVERY test.

mykola-mokhnach commented 1 month ago

The "device" is the one requested via DesiredOptions on SauceLabs. SauceLabs spins up a new instance for EVERY test.

Sorry, my question was a bit different. Do you always request the same device name from your test or every time the test is executed it runs on a different device from the cloud?

agray commented 1 month ago

Sorry, my question was a bit different. Do you always request the same device name from your test or every time the test is executed it runs on a different device from the cloud?

Same type of device, yes. How the selenium grid (which is what SauceLabs is) provides that device it up to the grid and part of the grid "service".

agray commented 1 month ago

https://github.com/Sauceforge/Saucery/commit/54da2c8d95ef1f5d6622e52c2aff68164cf0f79f

https://github.com/Sauceforge/Saucery/actions/runs/10338099541

appiumRequests.json appium.log device.log

mykola-mokhnach commented 1 month ago

Lets sum up what needs to be done step by step:

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)
Run the same test with an older appium snapshot via appiumVersion sauce option
Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session)
Run the same test on a different device

KazuCocoa commented 1 month ago

Hm, I don't have 17.4.1 but have 17.5.1 iPhone. It returned an empty safari view as below with browserName: safari, so I was not able to see pageArray: undefined case as attached in this ticket. I changed web inspector config as well though. The base xcuitest was 7.24.14. https://gist.github.com/KazuCocoa/4e2b14918e57cc1e3df932cc1cf789b3

Possibly the device has a specific condtion

agray commented 3 weeks ago

Any updates on this?

Dor-bl commented 3 weeks ago

Any updates on this?

Did you performed the below steps?

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)
Run the same test with an older appium snapshot via appiumVersion sauce option
Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session) Run the same test on a different device

mykola-mokhnach commented 3 weeks ago

I tend to think this is an issue in iOS itself. I was able to reproduce the same behaviour locally with iOS 17.4 and 17.5 Simulators via the Safari remote debugger. The only viable solution was to upgrade iOS version to 17.6 or 18.0

Btw, downgrade to 17.2 or 17.3 also did the trick

agray commented 3 weeks ago

Any updates on this?

Did you performed the below steps?

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)

Run the same test with an older appium snapshot via appiumVersion sauce option

Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session) Run the same test on a different device

Have you?

agray commented 3 weeks ago

Had to rollback to IPhone 14 Pro Max on IOS 16.2 on emulated (IPhone 15 Pro Max on IOS 17.0 just hangs) Had to rollback to IPhone 14 Pro Max on IOS 16 on real devices (IPhone 15 Pro Max on IOS 17 just hangs)

appium / dotnet-client