search

appium / appium

Cross-platform automation framework for all kinds of apps, built on top of the W3C WebDriver protocol
http://appium.io/
Apache License 2.0
18.86k stars 6.08k forks source link

Get timeout error when running test with Appium CLI #16056

Closed cvnboss closed 2 years ago

cvnboss commented 2 years ago

The problem

Get timeout error when running test with Appium CLI.

Environment

Details

Which the same capabilites, i got timeout error when running test with Appium CLI which not occur with Appium Server GUI. Can anyone help?

Link to Appium logs

CLI log:

GUI log:

Code To Reproduce Issue [ Good To Have ]

jlipps commented 2 years ago

It's unclear to me what could be going on. It's the same version of Appium and the same capabilities, right? And it reliably works on one but not the other? Seems like the logs are truncated so not sure what else to investigate.

cvnboss commented 2 years ago

It's the same version of Appium and the same capabilities, right? And it reliably works on one but not the other? Seems like the logs are truncated so not sure what else to investigate.

@jlipps Yes, with the same capabilities, both CLI and GUI are using appium 1.22.0, but it only works on GUI. The error log is not truncated, it's all the Appium CLI log provided (the GUI did because it showed a successful connection to WDA): Appium_CLI_Timeout_Error

KazuCocoa commented 2 years ago

I have no good idea, but I guess portforward or something did not work... Did the environment have proxy or something that might affect portforward?

PMLehmann commented 2 years ago

I have the same issue, could it be related to this change? https://github.com/appium/WebDriverAgent/pull/538

Using Appium 1.22.2

also: https://stackoverflow.com/questions/70298400/why-am-i-getting-an-xcuitest-failed-to-receive-any-data-within-the-timeout-5

KazuCocoa commented 2 years ago

Hm.., I don't think it is related to... since the change was to fix an error in Xcode 13 when useXctestrunFile is enabled added as Appium 1.22.1. Appium 1.22.0 does not have it.

What was your capabilities?

https://github.com/appium/appium-ios-device/blob/b47d5b394f884b597281ebabd48144e6fc17d23f/lib/plist-service/index.js#L67 is the log message.

PMLehmann commented 2 years ago

"appium:platformName": "iOS", "appium:platformVersion": "12.5.3", "appium:deviceName": "XXXXXXX", "appium:app": "com.apple.Preferences", (no matter which app) "appium:automationName": "XCUITest", "appium:udid": "XXXXXXXXXXXXXXX", "appium:xcodeOrgId": "YYYYYYY", "appium:xcodeSigningId": "ZZZZZZZ", "appium:updatedWDABundleId": "XXXXXXXXX" (WDA is installed successfully, Appium Log confirms that) "appium:newCommandTimeout": 60, "appium:webkitResponseTimeout": 60000, "appium:webviewConnectTimeout": 20000, "appium:commandTimeouts": "20000", "appium:showIOSLog": true, "appium:showXcodeLog": true, "appium:launchTimeout": 60000, "appium:skipLogCapture": false, "appium:clearSystemFiles": false, "appium:wdaLaunchTimeout": 60000, "appium:wdaConnectionTimeout": 60000,

I was trying to push every timeout up to astronomic numbers in order to try to get around the 5000ms. WDA is signed and installed, Appium is able to open a session. But device neither sends logs nor any data back.

PMLehmann commented 2 years ago

Adding Appium Log:

[BaseDriver] Session created with session id: 67239236-05fd-443b-8801-b43b0fce660f [debug] [XCUITest] Current user: 'xxxxxx' [debug] [XCUITest] Available devices: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [debug] [XCUITest] Creating iDevice object with udid 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' [XCUITest] Determining device to run tests on: udid: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', real device: true [XCUITest] Normalized platformVersion capability value '12.5.3' to '12.5' [debug] [BaseDriver] Event 'xcodeDetailsRetrieved' logged at 1641804189563 (09:43:09 GMT+0100 (Central European Standard Time)) [debug] [XCUITest] App is an iOS bundle, will attempt to run as pre-existing [debug] [BaseDriver] Event 'appConfigured' logged at 1641804189564 (09:43:09 GMT+0100 (Central European Standard Time)) [debug] [BaseDriver] Event 'resetStarted' logged at 1641804189564 (09:43:09 GMT+0100 (Central European Standard Time)) [debug] [XCUITest] Reset: running ios real device reset flow [debug] [BaseDriver] Event 'resetComplete' logged at 1641804189564 (09:43:09 GMT+0100 (Central European Standard Time)) [WebDriverAgent] Using WDA path: '/Users/xxxxxxx/.nvm/versions/node/v17.3.0/lib/node_modules/appium/node_modules/appium-webdriveragent' [WebDriverAgent] Using WDA agent: '/Users/xxxxxxx/.nvm/versions/node/v17.3.0/lib/node_modules/appium/node_modules/appium-webdriveragent/WebDriverAgent.xcodeproj' [debug] [WebDriverAgent] Parsed BUILD_DIR configuration value: '/Users/xxxxxxxxxx/Library/Developer/Xcode/DerivedData/WebDriverAgent-grzdpxobpiwlziequfyoysrpfuka/Build/Products' [debug] [WebDriverAgent] Got derived data root: '/Users/xxxxxxxxxx/Library/Developer/Xcode/DerivedData/WebDriverAgent-grzdpxobpiwlziequfyoysrpfuka' [XCUITest] Continuing without capturing device logs: Failed to receive any data within the timeout: 5000 [XCUITest] Setting up real device [XCUITest] {} [DevCon Factory] Releasing connections for xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx device on any port number [DevCon Factory] No cached connections have been found [debug] [XCUITest] Not clearing log files. Use clearSystemFiles capability to turn on. [debug] [BaseDriver] Event 'newSessionStarted' logged at 1641804199762 (09:43:19 GMT+0100 (Central European Standard Time)) [debug] [W3C] Encountered internal error running command: Error: Failed to receive any data within the timeout: 5000 [debug] [W3C] at Timeout._onTimeout (/Users/xxxxxxxxxxxxxxx/.nvm/versions/node/v17.3.0/lib/node_modules/appium/node_modules/appium-ios-device/lib/plist-service/index.js:67:16) [debug] [W3C] at listOnTimeout (node:internal/timers:568:17) [debug] [W3C] at processTimers (node:internal/timers:510:7) [HTTP] <-- POST /wd/hub/session 500 10416 ms - 671

jlipps commented 2 years ago

@PMLehmann are you running against a real device? seems like your idevicesyslog or whatever you're using to connect to the real device to retrieve logs isn't working.

PMLehmann commented 2 years ago

Yes, I am, tried on two different devices (both iPhone 6s). Yesterday I tracked the issue down to me being able to build, sign and run/test WebdriverAgent including WebdriveragentLib via the XCode GUI without any issues, but not over CLI with xcodebuild. Here I got a Signing error withWebdriveragentLib.Framework.

"command codesign failed with a nonzero exit code" and errSecInternalComponent.

I tried almost all of the solutions in https://stackoverflow.com/questions/52421999/xcode-10-command-codesign-failed-with-a-nonzero-exit-code but none worked so far.

I honestly don't get why I can build with XCode, but not XCodeBuild. And why Appium still creates a Session over CLI even though it is not working.

KazuCocoa commented 2 years ago

Your codesign was not a personal (free) one, right? Xcode could run the personal one somehow, but we got reports personal account did not work via xcodebuild for XCTest session. https://appium.io/docs/en/drivers/ios-xcuitest-real-devices/#troubleshooting-ideas

I wonder if you clean /Users/xxxxxxxxxx/Library/Developer/Xcode/DerivedData once

PMLehmann commented 2 years ago

I deleted the contents of Derived Data several times and also am using my companies account with Distribution rights.

Here is the relevant part of the xcodebuild-Log:

Copying /Users/xxxxxxxxxxxxxx/Library/Developer/Xcode/DerivedData/WebDriverAgent-grzdpxobpiwlziequfyoysrpfuka/Build/Products/Debug-iphoneos/WebDriverAgentLib.framework Testing failed: Command CodeSign failed with a nonzero exit code Testing cancelled because the build failed.

TEST FAILED

The following build commands failed: CodeSign /Users/xxxxxxxxxxx/Library/Developer/Xcode/DerivedData/WebDriverAgent-grzdpxobpiwlziequfyoysrpfuka/Build/Products/Debug-iphoneos/WebDriverAgentRunner-Runner.app/PlugIns/WebDriverAgentRunner.xctest/Frameworks/WebDriverAgentLib.framework (in target 'WebDriverAgentRunner' from project 'WebDriverAgent') (1 failure)

jlipps commented 2 years ago

Since Xcode uses xcodebuild under the hood, is there a way to get it to show you what xcodebuild command it generates for building? Then you could compare with what Appium is running to see if there are any discrepancies.

PMLehmann commented 2 years ago

Since I still made no progress on this: are there any known issues regarding Apples M1 Architecture & Appium or WDA? The problems seem to have started when we upgrades our Mac.

KazuCocoa commented 2 years ago

Afaik, no. I've tested Appium's some repositories' test suite (e.g. ruby's one, xcuitest-driver's one and WDA repository's one) on my M1 machine in the 1.20.0 release. https://github.com/appium/appium/blob/904798c40258d667b5ee932207a00d5000954d90/CHANGELOG.md#iosxcuitest-4

Potentially newer M1/macOS versions after that got some issues, but I haven't observed any M1 specific issues on my side yet.

jlipps commented 2 years ago

@PMLehmann I'm on an m1 as well, but no longer have an apple dev account so haven't tried a real device yet. Otherwise I haven't noticed any issues.

PMLehmann commented 2 years ago

Good news: an upgrade to Appium 2.0 and also the signing (same certificates etc.) of the webdriveragent found under /Users/xxxxxxx/.appium/appium-xcuitest-driver/node_modules/appium-webdriveragent/webdriveragent.xcodeproj seems to have fixed the issue. Don't know which of these two solved it.

Not happy with using the 2.0-Beta for this project, but at least it works for now. Thanks for your help so far!

KazuCocoa commented 2 years ago

Thank you for sharing the result.

I guess Xcode cache or xcodebuild's project handling stuff... The new project (via 2.0) was handled as a new one for the xcodebuild, so the build worked from clean build env...

KazuCocoa commented 2 years ago

Moves to https://github.com/appium/appium/issues/16399#issuecomment-1066054950 You need OPENSSL_TLS_SECURITY_LEVEL=0 built openssl 3 or openssl 1.1 as node.processes. We've confirmed openssl v3 did not work without https://github.com/appium/appium-ios-device/pull/88#discussion_r825315862 Simply, you can use opensslv1.