search

calabash / calabash-ios

Calabash for iOS
Other
1.81k stars 369 forks source link

CALABASH 0.20.0 : test have slowed down on ios 9.3.5 #1195

Open anshulsachan opened 7 years ago

anshulsachan commented 7 years ago

Calabash 0.20.0 : Running same set of test on ios 9.3.5 and ios 10.

Definitely see a speed up for ios10 but considerably slowed down for ios 9.3.5

ark-konopacki commented 7 years ago

@anshulsachan can you share some details? like tests run outputs for ios 10 and 9:

6 scenarios (6 passed)
95 steps (95 passed)
7m42.227s

and maybe you can identify on that kind of operation slowing down appears?

anshulsachan commented 7 years ago

Test output for ios 10 32 scenarios (32 passed) 559 steps (559 passed) Finished in 62m25.757s second

Test output for ios 9.3.5 32 scenarios (32 passed) 559 steps (559 passed) Finished in 104m31.178s seconds

Same scenarios were executed on both the devices from the same machine

JoeSSS commented 7 years ago

Are you sure that it is a slow down? For sure iOS 10 automation is way faster than old one, but question is if iOS 9 was faster before 0.20.0

anshulsachan commented 7 years ago

yes it has slowed down. Earlier before 0.20.0, i was executing the same scenarios on ios 9.3.2 and it would take around 75 mins to finish and now it is taking 104 mins

jmoody commented 7 years ago

Thank you for the analysis.

I assume this against simulators? If so, I wonder how much time is spent waiting for the simulator to launch.

ark-konopacki commented 7 years ago

From my setup only launching app in simulator 

DEBUG: Launching #<Simulator: iPhone 6 (9.3)
DEBUG: Waited for 7.38789 seconds for 'com.apple.audio.SystemSoundServer-iOS-Simulator' to start.
DEBUG: Waited a total of 27.767515 seconds for simulator to stabilize
DEBUG: Took 27.792618 seconds to launch the simulator
DEBUG: Waited a total of 7.169538 seconds for simulator to stabilize
DEBUG: Waited a total of 8.091466 seconds for simulator to stabilize
DEBUG: Took 11.594522 seconds to install DeviceAgent
DEBUG: DeviceAgent says, "Calabash is ready and waiting."
DEBUG: Took 15.232977 launch and respond to /health
DEBUG: Took 0.02056 seconds to check if app was installed
DEBUG: Took 15.365663 seconds to launch application
DEBUG: Launching #<Simulator: iPhone 6 (10.0)
DEBUG: Timed out after 30 seconds waiting for simulator to stabilize
DEBUG: Took 30.896232 seconds to launch the simulator
DEBUG: Waited for 10.057934 seconds for 'com.apple.audio.SystemSoundServer-iOS-Simulator' to start.
DEBUG: Waited a total of 30.388095 seconds for simulator to stabilize
DEBUG: Waited for 10.047862 seconds for 'com.apple.audio.SystemSoundServer-iOS-Simulator' to start.
DEBUG: Waited a total of 23.670462 seconds for simulator to stabilize
DEBUG: Took 28.101737 seconds to install DeviceAgent
DEBUG: DeviceAgent says, "Calabash is ready and waiting."
DEBUG: Took 31.838428 launch and respond to /health
DEBUG: Took 0.034434 seconds to check if app was installed
DEBUG: Took 31.990134 seconds to launch application

So for me it takes more time to launch iOS 10 simulator but i have some strange observation like entering text on iOS 9.3 takes a lot of time and sometimes results with HTTPClient::ReceiveTimeoutError will continue investigation

anshulsachan commented 7 years ago

mine analysis was against device, gen 6 ios 10 ipod and gen 5 ios 9.3.5 ipod

jmoody commented 7 years ago

Since there is nothing we can do about this, I am moving to the Icebox. Most users have seen a significant speed up over UIAutomation.