multi_scan does not find any tests when :only_testing is an array of testTargets

selin194 commented 4 years ago

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any gesture: https://github.com/sponsors/lyndsey-ferguson. 😍

Issue Description

When we upgrade test_center_plugin version to the latest ('3.13.2') tests are not run. It listed all of the tests but it shows following logs:

DEBUG [2020-08-14 11:59:26.81]: > setup_logcollection
DEBUG [2020-08-14 11:59:26.81]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2020-08-14 11:59:26.81]: Removing pre-existing xcresult bundles: 
DEBUG [2020-08-14 11:59:26.81]:   /Users/selinunal/Projects/mobile/ipad/fastlane/test_output/report.xcresult
DEBUG [2020-08-14 11:59:26.81]: Fetching available simulator devices
DEBUG [2020-08-14 11:59:27.07]: Results for each test run: []

+-------------------+------+
|    multi_scan results    |
+-------------------+------+
| result            | true |
| total_tests       | 0    |
| passing_testcount | 0    |
| failed_testcount  | 0    |
| failed_tests      | []   |
| total_retry_count | 0    |
| report_files      | []   |
+-------------------+------+

Complete output when running fastlane, including the stack trace and command used

[INSERT OUTPUT HERE]

Environment

I will list some gem versions below:

bundler            | 2.1.4
xcresult           | 0.2.1 
xctest_list        | 1.2.1

NOTE: Our tests are executed successfully using fastlane-plugin-test_center **3.13.0** version. Do you have any idea @lyndsey-ferguson? Thanks!

lyndsey-ferguson commented 4 years ago

@selin194 thank you for the bug report, sorry that this is happening.

Can you share how you are calling multi_scan? Two things changed since 3.13.0: 1) the method in which testsuites are expanded when given to the :only_testing option, and 2) how :parallel_testrun_count is pinned to the number of tests (i.e. if you only have 2 tests, and ask for 4 parallel test runs, you'll get 2 as 2 parallel test runs are wasted).

lyndsey-ferguson commented 4 years ago

I can see that tests are being picked up by these integration checks.

selin194 commented 4 years ago

Hey! We are calling multi_scan with only_testingand parallel_testrun_countwith value 2.

selin194 commented 4 years ago

Not sure why I can not see our tests are not executed 🤔

lyndsey-ferguson commented 4 years ago

Yes, but what values are in :only_testing? Is it an array of testsuites? An array of 'test-targets/testSuites'? An array of 'testSuites/testcases'?

selin194 commented 4 years ago

We are calling like only_testing: 'ourTests',

lyndsey-ferguson commented 4 years ago

And is that a testSuite?

selin194 commented 4 years ago

It is a test target under the app scheme. We have multiple test targets (unit, e2e, snapshot) and we are calling only 1 target at a time. Therefore we are using only_testing option by providing target name. Each target can have multiple test suites.

jambudi commented 4 years ago

@selin194 We have a similar design. I have ApplicationUI under our schema and many other different target and when I use only_testing that is not picking up our application tests and if I do not use only_testing, in that case, it is picking up something random unit tests. I tried to use scan and it works fine with scan

lyndsey-ferguson commented 4 years ago

Okay, that is something that I hadn't anticipated. It makes sense, for now, can you use 3.13.1 or earlier?

jambudi commented 4 years ago

I can try

jambudi commented 4 years ago

I think:only_tests doesn't work in 3.13.0

lyndsey-ferguson commented 4 years ago

@jambudi you may be experiencing a different problem from @selin194; she found that 3.13.0 did execute her tests successfully.

Can you run your fastlane with the --verbose flag and include the logs in this Issue as well as provide an example of how you are calling multi_scan (please obfuscate any sensitive data).

jambudi commented 4 years ago

I will open a different ticket. Thank you.

lyndsey-ferguson commented 4 years ago

@selin194 can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-271-only-testing-test-targets-not-expanded"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

selin194 commented 4 years ago

Hello @lyndsey-ferguson! I tried with this branch and it seems that it worked and all tests are executed. Will it be fixed? Thanks!

lyndsey-ferguson commented 4 years ago

@selin194 thank you for confirming 🎉 ! This has been released in v3.14.1

lyndsey-ferguson / fastlane-plugin-test_center