Code coverage final results after failing tests

[fabioknoedt]

Question Checklist

• [X] Updated fastlane-plugin-test_center to the latest version
• [X] I read the README.md
• [X] I reviewed the example(s) for the action(s) I am using
• [X] I searched for existing GitHub issues

Question Subject

Code coverage final results after failing tests

Question Description

I have the feeling there are similar questions but yet not the same. This issue was supposed to fix this but it still doesn't work for me: #33 So, I was wondering if the plugin is supposed to support the use case:

• First test run returns 98 out of 100 tests passed and code coverage of 85%
• Second test run (to retry the 2 failed tests) returns 2 out of 2 tests passed and code coverage of 5%
• Final code coverage that I get is: 5%

Is the plugin supposed to unify all test coverages and return 85% + ~5%?

Thanks!

[lyndsey-ferguson]

I attempted to resolve that issue as I mentioned here. Do you see multiple code coverage reports? What tool are you using to generate the final report?

[fabioknoedt]

Hey @lyndsey-ferguson, thanks for answering!

Do you see multiple code coverage reports? I am not sure as it is not printed in the logs. Is there an easy way for printing or inspecting the files created (i am running in the CI).

That's my Fastlane lane:

  desc "runs all the tests"
  lane :test do

    number_retries = 3

    test_run_block = lambda do |testrun_info|
      puts testrun_info
      failed_test_count = testrun_info[:failed].size
      if failed_test_count > 0
        UI.important('The run of tests would finish with failures due to fragile tests here.')
        try_attempt = testrun_info[:try_count]
        if try_attempt < number_retries
          UI.header('Since we are using :multi_scan, we can re-run just those failing tests!')
        end
      end
    end

    result = multi_scan(
      workspace: "Aiden.xcworkspace",
      try_count: number_retries,
      fail_build: false,
      devices: ["iPhone 8"],
      skip_slack: true,
      code_coverage: true,
      only_testing: ["AidenTests", "AidenUITests"],
      derived_data_path: "../DerivedData",
      output_types: "junit",
      disable_concurrent_testing: true,
      testrun_completed_block: test_run_block
    )

    trainer(path: "../DerivedData", output_directory: ".")
    slather(simple_output: true) # first display in the logs
    slather # then creates the file for Code Climate

  end

[lyndsey-ferguson]

I think that the way you have written your lane, the results will be in test_output directory. You should see multiple xccoverage files.

[fabioknoedt]

I just checked it locally and it is generating one single file called Coverage.profdata

[lyndsey-ferguson]

If I recall correctly, slather is responsible for converting xccoverage files to that profdata format. Can you comment out those two lines and see if it generates multiple xccoverage files?

[lyndsey-ferguson]

I could be wrong

[lyndsey-ferguson]

@fabioknoedtbcgdv have you had a chance to comment out the slather lines to see if we get xccoverage files?

[fabioknoedt]

@lyndsey-ferguson still no xccoverage file, only Coverage.profdata

[lyndsey-ferguson]

Ok, so probably your project is generating the code coverage reports in a way that I did not anticipate. You could modify your test_run_block lambda to rename the Coverage.profdata to Coverage-#{try_attempt} so that each test run does not overwrite the Coverage report. Then, when slather runs, it will handle all the files.

If you could attach a sample project that exhibits the same problem, I can take a look at fixing this issue in multi_scan itself.

[fabioknoedt]

Unfortunately, I can't attach my project but I could send you separated files if you tell me which ones you would look at. Also, I will try what you said. Are you sure Slather will combine the files instead of summing up coverages?

[lyndsey-ferguson]

I don't need your project, but if you could put together a sample project that has similar settings, and a Fastfile that runs multi_scan that generates the code coverage files in the same way that your project is. That would help me solve the issue quickly (rather than trying to intuit my way through it from some files).

From what I've seen, slather does combine the results, but I don't use the tool so I am only guessing from my experience in fixing the issue that I referenced earlier.

[fabioknoedt]

So, I cloned my project and removed everything and just left the configuration with some flaky tests. I hope it helps you debugging. Let me know if I can help further. And thanks a lot!

For setup: bundle install pod install fastlane ios test

example-testcenter.zip

You will see that even failing multiple times, only one file gets created Coverage.profdata and gets replaced in every multi_scan run.

[lyndsey-ferguson]

Thank you, I'll use this to investigate the issue. I have two other issues that I am working on in front of this.

[fabioknoedt]

Cool, thanks! I am here if you need anything else.

[fabioknoedt]

@lyndsey-ferguson hey, have you tried to run my attached project?

[lyndsey-ferguson]

No, not yet, I still have one other issue in front of this that I'm working on.

[jineshqa]

Hi @lyndsey-ferguson,

If I understand correctly, code coverage is considered only for first run. Subsequent runs do not collect code coverage report. Is that correct?

My project generates action.xccovreport file and directory "action.xccovarchive" for code coverage, these are generated under 1_Test directory. Is it possible to generate coverage under 2_Test directory for 2nd run and so on? I could copy all coverage files under one directory and do

xcrun xccov view Build/Logs/Test/*.xccovreport --json

Further process the json to get the overall code coverage and other info. Kindly let me know your thoughts.

Thanks, Jinesh

[jineshqa]

Also, this can be done to get one file.

xcrun xccov view reports/**/*.xccovreport > test.xccovreport

[lyndsey-ferguson]

@jineshqa are you seeing that there is a 2_Test directory? Or are you suggesting not turning off code coverage and creating a 2_Test directory?

[lyndsey-ferguson]

If you do see that there is such a directory, you could put some code in a testrun_completed_block option for multi_scan and perform the calls you outlined.

[jineshqa]

@lyndsey-ferguson

And people ask me why I like this plugin...speed of response...Thank you so much..

I am suggesting not turning off or providing a choice of not turning off code coverage and create 2_Test directory.

Currently, there is no 2_Test directory created and at the end all the results are merged and I see only 1 directory.

Thanks, Jinesh

[lyndsey-ferguson]

Hm. From what I remember when I worked with the code in the results_bundles (not sure of the name directly), xcodebuild creates those folders for you. I'd have to rename those directories.

[lyndsey-ferguson]

You can see that here: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb#L232

[lyndsey-ferguson]

I'd be willing to guide you on fixing this if you want to take it on? I still have one other issue that I want to fix first.

[jineshqa]

I've not used Ruby at all but I can try. Also, while reading code I noticed this line, which indicates that you are deleting code coverage option on the subsequent run.

[lyndsey-ferguson]

Right, that's where I "turn it off".

[jineshqa]

Also, If I comment this line under function collate_test_result_bundles _testcenter would not delete the result bundle directories at end of result collation. Then I and others could converge coverage files to get full coverage. Is that right?

[lyndsey-ferguson]

That's right, but the "collation" should bring everything together and those other directories are redundant.

[jineshqa]

Below shell command can collate the coverage files. I am not sure if we have anything in fastlane/Ruby to do that. Is it OK to call a shell command from collate function?

xcrun xccov view reports/**/*.xccovreport > test.xccovreport

[jineshqa]

See line 5 from bottom starting with exec

def collate_test_result_bundles(output_directory, reportnamer)
        test_result_bundlepaths = Dir.glob("#{output_directory}/#{@scan_options[:scheme]}*.test_result").map do |relative_test_result_bundle_filepath|
          File.absolute_path(relative_test_result_bundle_filepath)
        end
        if test_result_bundlepaths.size > 1
          config = FastlaneCore::Configuration.create(
            Fastlane::Actions::CollateTestResultBundlesAction.available_options,
            {
              bundles: test_result_bundlepaths.sort { |f1, f2| File.mtime(f1) <=> File.mtime(f2) },
              collated_bundle: File.join(output_directory, @scan_options[:scheme]) + '.test_result'
            }
          )
          Fastlane::Actions::CollateTestResultBundlesAction.run(config)
          exec( "xcrun xccov view #{output_directory}/**/*.xccovreport > #{@scan_options[:scheme]}.xccovreport" )
        end
        retried_test_result_bundlepaths = Dir.glob("#{output_directory}/#{@scan_options[:scheme]}_*.test_result")
        FileUtils.rm_rf(retried_test_result_bundlepaths)
      end

[lyndsey-ferguson]

Actually, that would be better done in the CollateTestResultBundlesAction method here: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb#L19

You can call the shell command similar to this: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb#L69

[lyndsey-ferguson]

What happens if there are no xccovreport files and you send it to xcrun xccov view?

[jineshqa]

Could you kindly explain why should the change be made at https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/actions/collate_test_result_bundles.rb#L19? Is CollateTestResultBundlesAction automatically called at the end of test run?

If there are no xccovreport file we get an error, so I need to check if @scan_options[:code_coverage] is set to true or not.

[lyndsey-ferguson]

Fastlane::Actions::CollateTestResultBundlesAction is responsible for the collation, which your shell command is doing. I'd rather have the logic for collation in one place. You can see the action called on the line above your shell command example.

[lyndsey-ferguson]

As this shell command is only valid if the files exist, you can add a check. I will add an example in a moment

[jineshqa]

Thanks, understood. Let me fork the repo and take a stab.

[lyndsey-ferguson]

At the beginning of the collate_test_result_bundles.rb:

require 'shellwords'

Later where I suggested:

xccovreports = []
test_result_bundlepaths.each do |test_result_bundlepath|
  xccovreports.concat(Dir.glob("#{test_result_bundlepath}/**/*.xccovreport"))
end
unless xccovreports.empty?
  bundle_name = File.basename(params[:collated_bundle], '.*')
  bundle_dir = File.dirname(params[:collated_bundle])
  sh("xcrun xccov view #{xccovreports.shelljoin} > #{bundle_dir}/#{bundle_name}.xccovreport ", print_command: false, print_command_output: false)
end

[lyndsey-ferguson]

@jineshqa you may find this document helpful too

[jineshqa]

@lyndsey-ferguson I was wrong about xcrun xccov view reports/**/*.xccovreport > test.xccovreport command or there is a bug. Either way the result file, test.xccovreport, is not readable. I tried to not delete the redundant directories and generate coverage data, but coverage report generated is also incorrect. Same set of tests produce different coverage when run from xcode.

Let me know if you have any other ideas.

Thanks, Jinesh

[lyndsey-ferguson]

What if you converted the xccovreport to json for each test_bundle and then collate the json? Would a final json report give you what you want? Or, does the "view" command only display a GUI with the report (which not help in this case)?

Otherwise, we would have to parse the report ourselves.

[lyndsey-ferguson]

What if you use the gem https://github.com/nakiostudio/xcov to convert the report file to json or html and then collate the results?

[jineshqa]

Let me try using xcov, if it is able to handle multiple xccovreport files, then we may be able to use it.

[lyndsey-ferguson]

Even if it cannot, you can still process each xccovreport file to create a separate json, and the individual json files can be collated.

[jineshqa]

Ok..so xcov hardcodes the coverage file location. I have created an issue with them. Let's see

https://github.com/nakiostudio/xcov/issues/142

[lyndsey-ferguson]

If you run the xcov tool after each run to convert the file to json, you should be able to collect each one after the entire multi_scan is complete and collate them.

[jineshqa]

You mean xcov or xccov?

[jineshqa]

Also collating json file in this case would be a challenge. For example lets say that we ran 2 test cases which covers same class file. In the first run 1 test case passed and 1 failed. So now code coverage for both the runs may have some line coverage overlap. So we'll need to identify and recalculate line coverage accordingly, we can't simply add.

[jineshqa]

@lyndsey-ferguson There is something else that I observed about the original bug logged i.e. issue#87. When tests run 2nd time original test bundle directory is renamed. For example lets say test bundle is named as test_bundle.test_result during 2nd run this directory is renamed to test_bundle_1.test_result. 2nd run bundle is named as test_bundle.test_result, which is then considered as base bundle in collation code. Since there is no handling for coverage file it is never copied from _1 to this base folder and hence lost. Which is the behavior mentioned in this issue. I am not an expert but this is what I think is happening please correct me if I am missing something.

[lyndsey-ferguson]

I think that you could be correct. I have not built in technology to "collate" the coverage reports, so the only report that exists is the one that was generated in the 1st bundle. That's where the suggestion I made may work.

[fabioknoedt]

hi @jineshqa! did you make it to work?