Running on Specific Devices

fatahillahardhi commented 8 months ago

Hello Prashant, I'd like to execute a somewhat specific scenario.

I have a test on iOS with a scenario that involves capturing with a camera, and it can only be done on a Real Device. However, currently, I'm running the test on both a Real Device and a Simulator simultaneously. Now, I want to ensure that when running a scenario containing the tags @UploadPhotos, it should only be executed on a Real Device and not on the Simulator. Is there a solution for this? Thank you.

prashant-ramcharan commented 8 months ago

Hi @fatahillahardhi

As this is not currently supported, I will investigate if we can differentiate between simulators and real devices in the same run.

fatahillahardhi commented 8 months ago

Hi @prashant-ramcharan , hope you're doing well,

Thank you for the answer. I'm very interested to wait for the updates!

prashant-ramcharan commented 7 months ago

@fatahillahardhi This is now supported in Courgette version 6.11.0

Usage example

Courgette will only allocate a real device to tests matching any of the tags in realMobileDeviceTag. For all other tests, one of the simulators will be allocated.

@RunWith(Courgette.class)
@CourgetteOptions(
        ...
        plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
        mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
        mobileDevice = {
                "simulator1",
                "simulator2"
                "realDevice:00000000-000-0000-0000-000000000001",
        },
        realMobileDeviceTag = {"@UploadPhotos"},
)

fatahillahardhi commented 7 months ago

Hi @prashant-ramcharan, it's really awesome!

However, I have encountered a problem. When I try to run it with SIMULATOR_AND_REAL_DEVICES, it always runs on Real Devices.

Here are my Test Runner:

@RunWith(Courgette.class)
@CourgetteOptions(
        runLevel = CourgetteRunLevel.SCENARIO,
        reportTargetDir = "build",
        testOutput = CourgetteTestOutput.CONSOLE,
        environmentInfo = "project=iOS Test",
        plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
        mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
        mobileDevice = {
                "iPhone 14",
                "iPhone Real:xxxxxxx-xxxxxxx"
        },
        realMobileDeviceTag = {"@UploadPhotos"},
        cucumberOptions = @CucumberOptions(
                features = "src/test/resources/features",
                publish = true,
                plugin = {
                        "pretty",
                        "json:build/cucumber.json"}
        ))

When I run the command: ./gradlew clean test -Dcourgette.threads=2 -Dcucumber.tags="@focus" -Dcourgette.rerunFailedScenarios=false, it consistently runs on Real Devices.

Moreover, when I try to print CourgetteMobileDeviceAllocator.DEVICE_NAME to the terminal, it shows "iPhone 14," but in reality, the execution occurs on Real Devices, and it doesn't run in parallel.

prashant-ramcharan commented 7 months ago

Hi, if you remove the realMobileDeviceTag option from your runner, does all tests run on the simulator?

Also, could you please share how your @UploadPhotos scenario is tagged?

Is it tagged on scenario level, scenario outline (example table) level or at the feature level?

prashant-ramcharan commented 7 months ago

I tested with this

@RunWith(Courgette.class)
@CourgetteOptions(
        threads = 3,
        runLevel = CourgetteRunLevel.SCENARIO,
        reportTargetDir = "build",
        testOutput = CourgetteTestOutput.CONSOLE,
        environmentInfo = "app=iOS test application; project_info=Courgette-JVM is awesome!",
        plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
        mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
        mobileDevice = {
                "iPhone 15",
                "iPhone 15:123",
        },
        realMobileDeviceTag = { "@upload" },
        cucumberOptions = @CucumberOptions(
                features = "src/test/resources/features",
                glue = "steps",
                tags = "@ios",
                publish = true,
                plugin = {
                        "pretty",
                        "json:build/cucumber-report/cucumber.json",
                        "html:build/cucumber-report/cucumber.html"}
        ))
public class IosTestRunner {
}

@ios
Feature: Test iOS application

  Scenario: Verify application alert on device 1
    Given I launch the app
    When I show the alert
    Then I verify the alert shows this alert is so cool.
    And I accept the alert

  @upload
  Scenario: Verify application alert on device 2
    Given I launch the app
    When I show the alert
    Then I verify the alert shows this alert is so cool.
    And I accept the alert

Output was as expected: 1 passed on the simulator and 1 failed (test tagged with @upload) because the real device does not exist.

───────────────────────────────────────────────────
             Courgette Test Statistics          
───────────────────────────────────────────────────
Summary:    50% passed, 50% failed
Duration:   0 min, 27 sec
Run Level:  Scenario
Total:      2
Passed:     1
Failed:     1
───────────────────────────────────────────────────

fatahillahardhi commented 7 months ago

Hi @prashant-ramcharan,

I have tried several tests with several cases:

Devices:

1 Simulator
1 Real Device

Case 1: Run with 2 scenarios, one of which contains the @UploadPhotos tag. Results:

Automation test does not run in parallel ❌
Runs on both Simulator and Real Devices ✅
Scenario with the @UploadPhotos tag does not run on the Simulator ✅

Case 2: Run with 3 scenarios, one of which contains the @UploadPhotos tag, and remove the realMobileDeviceTag. Results:

Automation test does not run in parallel ❌
Only runs on the Simulator ❌
Scenario with the @UploadPhotos tag runs on the Simulator ❌

Case 3: Run with 3 scenarios without the @UploadPhotos tag and remove the realMobileDeviceTag. Results:

Automation test does not run in parallel ❌
Only runs on the Simulator ❌
Scenario with the @UploadPhotos tag runs on the Simulator ❌

Case 4: Run with 3 scenarios without the @UploadPhotos tag and with realMobileDeviceTag.

Automation test does not run in parallel ❌
Only runs on the Simulator ❌
Scenario with the @UploadPhotos tag runs on the Simulator ❌

Overall I think realMobileDeviceTag is doing well, but the issues is the test do not run in parallel but instead run sequentially, similar to conventional testing.

Scenario:

@iOS @Example

  Feature: Scenario with Upload Tags

    @Test1
    Scenario: Login with already registered phone number
      Given User is on screen
      When User click button login
      And User insert phone number "0823141284"
      And User input correct OTP
      Then User is on bibit main page

    @Test2 @UploadPhotos
    Scenario: Login with unregistered phone number
      Given User is on screen
      When User click button login
      And User insert phone number unregistered "081231231231"
      Then User direct otp page without get the otp

    @Test3 @UploadPhotos
    Scenario: Login with unregistered phone number 2
      Given User is on screen
      When User click button login
      And User insert phone number unregistered "081231231231"
      Then User direct otp page without get the otp

Test Runner

@RunWith(Courgette.class)
@CourgetteOptions(
        threads = 2,
        runLevel = CourgetteRunLevel.SCENARIO,
        rerunFailedScenarios = true,
        rerunAttempts = 1,
        reportTargetDir = "build",
        testOutput = CourgetteTestOutput.CONSOLE,
        environmentInfo = "project=iOS Test",
        plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
        mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
        mobileDevice = {
                "iPhone:xxxxxxx-xxxxxxx",
                "iPhone 14"
        },
        realMobileDeviceTag = {"@UploadPhotos"},
        cucumberOptions = @CucumberOptions(
                features = "src/test/resources/features",
                tags = "@Example",
                publish = true,
                plugin = {
                        "pretty",
                        "json:build/cucumber.json"}
        ))

prashant-ramcharan commented 7 months ago

Hi, with regards to this point:

Automation test does not run in parallel

You would need to specify more than 1 simulator or device to be able to run in parallel as Courgette can only allocate 1 device per run at a time.

We wont be able to run multiple tests in parallel if we only have 1 device / simulator per run.

Example:

 threads = 3,
 mobileDevice = {
                "iPhone:xxxxxxx-xxxxxxx",
                "iPhone 14",
                "iPhone 15"
                "iPhone 15 Pro"
        },

The above will run 3 tests in parallel on the simulators and only run tests on the real device if it matches the realMobileDeviceTag

Did you try adding more simulators / devices to see it runs in parallel?

Could you also try running the example project and let me know how you get on?

https://github.com/prashant-ramcharan/courgette-jvm-appium-ios

fatahillahardhi commented 7 months ago

Hi @prashant-ramcharan,

I have just tried adding more devices, but the result remains the same

Automation test does not run in parallel

mobileDeviceType = MobileDeviceType.SIMULATOR_AND_REAL_DEVICE,
mobileDevice = {
                "iPhone 14",
                "iPhone 1:00008020-00146850340B402E",
                "iPhone 2:00008020-001531EA2E22402E",
                "iPhone 4:00008030-001419043428C02E",
                "iPhone 6:3c240073ed0e0ecf6f786ebed31a9ee0b147e8bf",
},
realMobileDeviceTag = {"@UploadPhotos"}

prashant-ramcharan commented 7 months ago

Could you please try using just simulators to test?

Are you able to provide a demo project to reproduce this issue?

The tests run fine in parallel for me.. Link to video here

fatahillahardhi commented 7 months ago

Oh sorry, I forgot to tell you.

If I use MobileDeviceType.SIMULATOR or MobileDeviceType.REAL_DEVICE, it can run properly (run with parallel), but if I use MobileDeviceType.SIMULATOR_AND_REAL_DEVICE, the Automation test does not run in parallel.

prashant-ramcharan commented 7 months ago

Hi @fatahillahardhi , no worries 🙂

I'm also using MobileDeviceType.SIMULATOR_AND_REAL_DEVICE and it runs fine in parallel.

Please see video demo here

As you can see in the video, the tests ran in parallel in the simulators and failed when running on the real device because the device did not exist (as expected)

If you want to share your Appium logs during the run, I can take a look. Might be worth re-building / signing your web driver agent again on the real devices.

Not sure how to reproduce this, so please provide a demo project if possible.

fatahillahardhi commented 7 months ago

Hi @prashant-ramcharan,

I attempted to run the example project here and encountered a unique result.

I used both MobileDeviceType.SIMULATOR and MobileDeviceType.SIMULATOR_AND_REAL_DEVICE. Surprisingly, the results were the same whether I ran the tests with 3 simulators or with 3 simulators + 1 real device. Unfortunately, the automation tests did not run in parallel.

Case 1, when using either MobileDeviceType.SIMULATOR or MobileDeviceType.SIMULATOR_AND_REAL_DEVICE and running with 3 simulators or 3 simulators + 1 real device:

The automation test was expected to run in parallel but encountered errors.
Out of 4 scenarios, only 1 scenario passed, while the other 3 failed consistently after multiple attempts.
You can find the logs here (running with simulators only).

Case 2, when using MobileDeviceType.SIMULATOR and running with 1 simulator:

All scenarios passed successfully.
You can find the logs here.

I hope this information helps in identifying and resolving the issues.

Test Runner:

@RunWith(Courgette.class)
@CourgetteOptions(
        threads = 3,
        runLevel = CourgetteRunLevel.SCENARIO,
        reportTargetDir = "build",
        testOutput = CourgetteTestOutput.CONSOLE,
        environmentInfo = "app=iOS test application; project_info=Courgette-JVM is awesome!",
        plugin = {CourgettePlugin.MOBILE_DEVICE_ALLOCATOR},
        mobileDeviceType = MobileDeviceType.SIMULATOR,
        mobileDevice = {
                "iPhone 15",
                "iPhone 15 Plus",
                "iPhone 15 Pro Max",
        },
        cucumberOptions = @CucumberOptions(
                features = "src/test/resources/features",
                glue = "steps",
                tags = "@ios",
                publish = true,
                plugin = {
                        "pretty",
                        "json:build/cucumber-report/cucumber.json",
                        "html:build/cucumber-report/cucumber.html"}
        ))

Features:

@ios
Feature: Test iOS application

  @Test1 @UploadPhotos
  Scenario: Verify application alert on device 1
    Given I launch the app
    When I show the alert
    Then I verify the alert shows this alert is so cool.
    And I accept the alert

  @Test2
  Scenario: Verify application alert on device 2
    Given I launch the app
    When I show the alert
    Then I verify the alert shows this alert is so cool.
    And I accept the alert

  @Test3
  Scenario: Verify application alert on device 3
    Given I launch the app
    When I show the alert
    Then I verify the alert shows this alert is so cool.
    And I accept the alert

  @Test4 @UploadPhotos
  Scenario: Verify application alert on device 4
    Given I launch the app
    When I show the alert
    Then I verify the alert shows this alert is so cool.
    And I accept the alert

prashant-ramcharan commented 7 months ago

I'm using the same project as shown in the videos above so cannot replicate your issue.

Are you able to provide any example project for me to reproduce this? I cannot reproduce this with the Courgette example project as shown in the videos.

Case 1, when using either MobileDeviceType.SIMULATOR or MobileDeviceType.SIMULATOR_AND_REAL_DEVICE and running with 3 simulators or 3 simulators + 1 real device:

Could you please share your Courgette html report? This should show any exceptions thrown in the test.

prashant-ramcharan commented 7 months ago

@fatahillahardhi In the mean time, I will release a new Courgette option to log the run details for each parallel run.

prashant-ramcharan commented 7 months ago

@fatahillahardhi I just released Courgette 6.12.0 with a new courgette option generateCourgetteRunLog

Could you try updating your runner and include the following option?

generateCourgetteRunLog = true

At the end of the test run you should have a courgette run log saved to your report directory.

Please share the run log and Courgette html report if you still want me to look into this.

fatahillahardhi commented 7 months ago

hi @prashant-ramcharan , sorry for the wait, I can only get back into this project now.

I have reattempted with some cases with version 6.12.0

Case 1: Simulator & Real Devices using realMobileDeviceTag @UploadPhotos

2 Simulators and 1 Real Device
Initially, only the 2 simulators with tags not containing @UploadPhotos are executed in parallel.
Once these 2 simulators are running, the real device does not react at all, more like waiting for the scenario without @UploadPhotos to finish.
When the @UploadPhotos scenario finishes, with only scenarios containing @UploadPhotos remaining, the Real Device starts running, resulting in no parallel execution (with Real Devices). Is this expected?

Case 2: Simulator & Real Devices not using realMobileDeviceTag @UploadPhotos

2 Simulators and 1 Real Device
Only simulators run in parallel; the real device does not run.
Everything else remains as expected.

Here's the report: Link

Update: I also added the video on that link

prashant-ramcharan commented 7 months ago

Hello @fatahillahardhi

Case 1

This is as expected.

When using MobileDeviceType.SIMULATOR_AND_REAL_DEVICE, Courgette calculates the max number of device types per type and orders the runs based on this.

The reason is Courgette cannot pre-allocate devices without actually knowing the scenario that's about to run next. So Courgette determines the strategy before any tests can run so it knows how to allocate devices based on the provided devices types. If you have more devices of a certain type then this will be used to determine the run order.

So for example, if you provide:

3 simulators and 1 real device

All scenarios without a matching realMobileDeviceTag will first run on simulators in parallel. Then all scenarios that require a real device (i.e. the scenario that has a matching realMobileDeviceTag) will run next.

3 real devices and 1 simulator

All scenarios that has a matching realMobileDeviceTag will first run on real devices in parallel. Then all scenarios that require a simulator will run next.

Case 2

If you use MobileDeviceType.SIMULATOR_AND_REAL_DEVICE and provide both simulators and real devices without specifying any matching realMobileDeviceTag then all scenarios will run on simulators.

You need to be explicit on which tests can + should run on real devices. Courgette does not know whether all tests can run on both simulators + real devices so it defaults to simulators.

If you don't specify a realMobileDeviceTag then you better of either using MobileDeviceType.SIMULATOR or MobileDeviceType.REAL_DEVICE

prashant-ramcharan / courgette-jvm

Running on Specific Devices #391