zebrunner / appium

Enhanced Appium for Elastic Selenium Grid/MCloud services
8 stars 3 forks source link

android offline state detection seems broken #327

Closed vdelendik closed 6 months ago

vdelendik commented 8 months ago

we have to review offline state detection as it seems broken and as result we don't try with reconnect

available: 0
unauthorized: 0
offline: 0
01-22 23:23:56.782    90   326 I adb     : transport.cpp:335 R5..4X: read thread spawning
01-22 23:23:56.782    90   327 I adb     : transport.cpp:307 R5..4X: write thread spawning
01-22 23:23:56.791    90   326 I adb     : transport.cpp:339 R5..4X: read failed: Success
01-22 23:23:56.791    90   326 I adb     : transport.cpp:1253 R5..4X: connection terminated: read failed
01-22 23:23:56.791    90    90 I adb     : adb.cpp:176 R5..4X: already offline
01-22 23:23:56.791    90    90 I adb     : transport.cpp:936 destroying transport R5..4X
01-22 23:23:56.791    90    90 I adb     : transport.cpp:407 BlockingConnectionAdapter(R5..4X): stopping
01-22 23:23:56.791    90   327 E adb     : transport_usb.cpp:166 remote usb: 1 - write terminated: Connection timed out
01-22 23:23:56.791    90    90 I adb     : transport.cpp:425 BlockingConnectionAdapter(R5..4X): stopped
01-22 23:23:56.791    90    90 I adb     : transport.cpp:294 BlockingConnectionAdapter(R5..4X): destructing
01-22 23:23:56.791    90    90 I adb     : transport.cpp:400 BlockingConnectionAdapter(R5..4X): already stopped
01-22 23:23:57.770    90   328 I adb     : transport.cpp:335 R5..4X: read thread spawning
01-22 23:23:57.771    90   328 I adb     : transport.cpp:339 R5..4X: read failed: Success
01-22 23:23:57.771    90   328 I adb     : transport.cpp:1253 R5..4X: connection terminated: read failed
01-22 23:23:57.771    90    90 I adb     : adb.cpp:176 R5..4X: already offline
01-22 23:23:57.771    90    90 I adb     : transport.cpp:936 destroying transport R5..4X
01-22 23:23:57.771    90    90 I adb     : transport.cpp:407 BlockingConnectionAdapter(R5..4X): stopping
01-22 23:23:57.772    90   329 I adb     : transport.cpp:307 R5..4X: write thread spawning
01-22 23:23:57.772    90    90 I adb     : transport.cpp:425 BlockingConnectionAdapter(R5..4X): stopped
01-22 23:23:57.772    90    90 I adb     : transport.cpp:294 BlockingConnectionAdapter(R5..4X): destructing
01-22 23:23:57.772    90    90 I adb     : transport.cpp:400 BlockingConnectionAdapter(R5..4X): already stopped
01-22 23:23:58.770    90   330 I adb     : transport.cpp:335 R5..4X: read thread spawning
01-22 23:23:58.771    90   331 I adb     : transport.cpp:307 R5..4X: write thread spawning
01-22 23:23:58.771    90   330 I adb     : transport.cpp:339 R5..4X: read failed: Success
01-22 23:23:58.771    90   330 I adb     : transport.cpp:1253 R5..4X: connection terminated: read failed
01-22 23:23:58.771    90    90 I adb     : adb.cpp:176 R5..4X: already offline
01-22 23:23:58.771    90    90 I adb     : transport.cpp:936 destroying transport R5..4X
01-22 23:23:58.771    90    90 I adb     : transport.cpp:407 BlockingConnectionAdapter(R5..4X): stopping
01-22 23:23:58.771    90   331 E adb     : transport_usb.cpp:166 remote usb: 1 - write terminated: Connection timed out
01-22 23:23:58.772    90    90 I adb     : transport.cpp:425 BlockingConnectionAdapter(R5..4X): stopped
01-22 23:23:58.772    90    90 I adb     : transport.cpp:294 BlockingConnectionAdapter(R5..4X): destructing
01-22 23:23:58.772    90    90 I adb     : transport.cpp:400 BlockingConnectionAdapter(R5..4X): already stopped
01-22 23:23:59.771    90   341 I adb     : transport.cpp:335 R5..4X: read thread spawning
01-22 23:23:59.771    90   341 I adb     : transport.cpp:339 R5..4X: read failed: Success
01-22 23:23:59.771    90   341 I adb     : transport.cpp:1253 R5..4X: connection terminated: read failed
01-22 23:23:59.771    90    90 I adb     : adb.cpp:176 R5..4X: already offline
01-22 23:23:59.772    90    90 I adb     : transport.cpp:936 destroying transport R5..4X
01-22 23:23:59.772    90    90 I adb     : transport.cpp:407 BlockingConnectionAdapter(R5..4X): stopping
01-22 23:23:59.771    90   342 I adb     : transport.cpp:307 R5..4X: write thread spawning
01-22 23:23:59.772    90    90 I adb     : transport.cpp:425 BlockingConnectionAdapter(R5..4X): stopped
01-22 23:23:59.772    90    90 I adb     : transport.cpp:294 BlockingConnectionAdapter(R5..4X): destructing
01-22 23:23:59.772    90    90 I adb     : transport.cpp:400 BlockingConnectionAdapter(R5..4X): already stopped
01-22 23:24:00.771    90   344 I adb     : transport.cpp:307 R5..4X: write thread spawning
01-22 23:24:00.771    90   343 I adb     : transport.cpp:335 R5..4X: read thread spawning
01-22 23:24:00.772    90   343 I adb     : transport.cpp:339 R5..4X: read failed: Success
01-22 23:24:00.772    90   343 I adb     : transport.cpp:1253 R5..4X: connection terminated: read failed
01-22 23:24:00.772    90    90 I adb     : adb.cpp:176 R5..4X: already offline
01-22 23:24:00.772    90    90 I adb     : transport.cpp:936 destroying transport R5..4X
01-22 23:24:00.772    90    90 I adb     : transport.cpp:407 BlockingConnectionAdapter(R5..4X): stopping
01-22 23:24:00.772    90    90 I adb     : transport.cpp:425 BlockingConnectionAdapter(R5..4X): stopped
01-22 23:24:00.772    90    90 I adb     : transport.cpp:294 BlockingConnectionAdapter(R5..4X): destructing
01-22 23:24:00.772    90    90 I adb     : transport.cpp:400 BlockingConnectionAdapter(R5..4X): already stopped
Device is not available!
Exiting without restarting...

root@17c53ce4e8ed:/home/androidusr# adb devices
List of devices attached
R5..4X     offline
vdelendik commented 8 months ago

this line doesn't cover above use-case https://github.com/zebrunner/appium/blob/e9036ee2ee16e2bff24f1d8625fd14acfe7a0db5/files/healthcheck#L18

@azarouski, as only identify and fix let's test together as adb reconnect seems recreate only appium and made stf containre unhealthy

azarouski commented 7 months ago

To test this bug you need to try to connect with device in different states You can use public.ecr.aws/zebrunner/appium:2.0.12-beta2 image for testing

vdelendik commented 7 months ago

positive scenario works great. closing...

vdelendik commented 6 months ago

reopening as this offline state we have to detect as part of healthcheck too. we have a problem with commands order here: https://github.com/zebrunner/appium/blob/4b0adf1aac1b35e4bd0d5b1e8fe384a721d9786c/files/healthcheck#L17

let's move device state detection onto the top, i case of offline try to reconnect. if not -> kill appium node process. Idea to exit with error code 0

vdelendik commented 6 months ago

device state detection fix was in registering stderr in the state variable here: https://github.com/zebrunner/appium/blob/1c73e8e5887d19ff7f506fff745e34fe2d3076b7/device_connect.sh#L86

state=$(adb get-state)
->
state=$(adb get-state 2>&1)

changes are available for testing in appium:2.0.14 because I already applied changes on one of the prod as a hot fix

vdelendik commented 6 months ago

reopening to verify as on startup so as part of healtcheck. feature might be implemented by moving get state into independent shell script: https://github.com/zebrunner/appium/blob/74b280207419623fbbec79a58a37ba46ea15dbf5/device_connect.sh#L85

in this case we can launch it as part of adb connect ... for wireless and for single device connected via usb.

vdelendik commented 6 months ago

@azarouski, that's a key state for the usbreset usage:

03-18 11:30:06.908    78    78 I adb     : auth.cpp:416 adb_auth_init...
03-18 11:30:06.908    78    78 I adb     : auth.cpp:152 loaded new key from '/root/.android/adbkey' with fingerprint B96B964CE463E3F56A992E030BA6DCD75264EA79C9665262EC121240BD0B9B92
03-18 11:30:06.908    78    78 I adb     : auth.cpp:391 adb_auth_inotify_init...
03-18 11:30:06.908    78    85 I adb     : transport.cpp:335 R..UN: read thread spawning
03-18 11:30:06.908    78    86 I adb     : transport.cpp:307 R..UN: write thread spawning
03-18 11:30:06.908    78    85 I adb     : transport.cpp:339 R..UN: read failed: Success
03-18 11:30:06.908    78    85 I adb     : transport.cpp:1253 R..UN: connection terminated: read failed
03-18 11:30:06.908    78    78 I adb     : adb.cpp:176 R..UN: already offline
03-18 11:30:06.908    78    78 I adb     : transport.cpp:936 destroying transport R..UN
03-18 11:30:06.908    78    78 I adb     : transport.cpp:407 BlockingConnectionAdapter(R..UN): stopping
03-18 11:30:06.908    78    86 E adb     : transport_usb.cpp:166 remote usb: 1 - write terminated: Connection timed out
03-18 11:30:06.908    78    78 I adb     : transport.cpp:425 BlockingConnectionAdapter(R..UN): stopped
03-18 11:30:06.908    78    78 I adb     : transport.cpp:294 BlockingConnectionAdapter(R..UN): destructing
03-18 11:30:06.908    78    78 I adb     : transport.cpp:400 BlockingConnectionAdapter(R..UN): already stopped
03-18 11:30:07.908    78    89 I adb     : transport.cpp:335 R..UN: read thread spawning
03-18 11:30:07.908    78    90 I adb     : transport.cpp:307 R..UN: write thread spawning
03-18 11:30:07.908    78    89 I adb     : transport.cpp:339 R..UN: read failed: Success
03-18 11:30:07.908    78    89 I adb     : transport.cpp:1253 R..UN: connection terminated: read failed
03-18 11:30:07.908    78    78 I adb     : adb.cpp:176 R..UN: already offline
03-18 11:30:07.908    78    78 I adb     : transport.cpp:936 destroying transport R..UN
03-18 11:30:07.908    78    78 I adb     : transport.cpp:407 BlockingConnectionAdapter(R..UN): stopping
03-18 11:30:07.908    78    90 E adb     : transport_usb.cpp:166 remote usb: 1 - write terminated: Connection timed out
03-18 11:30:07.908    78    78 I adb     : transport.cpp:425 BlockingConnectionAdapter(R..UN): stopped
03-18 11:30:07.908    78    78 I adb     : transport.cpp:294 BlockingConnectionAdapter(R..UN): destructing
03-18 11:30:07.908    78    78 I adb     : transport.cpp:400 BlockingConnectionAdapter(R..UN): already stopped
**state: error: no devices/emulators found**

it means we have device connected via usb because container exists but it is not visible by adb... tested successfully on different instances

azarouski commented 6 months ago

Please, use 2.0.15-beta1 image for testing

vdelendik commented 6 months ago

verified successfully on one of the production env/device.