search

fastlane-community / trainer

Convert xcodebuild plist and xcresult files to JUnit reports
https://krausefx.com
MIT License
248 stars 50 forks source link

Support for Xcode 11 #32

Closed thibault-ml closed 5 years ago

thibault-ml commented 5 years ago

Xcode 11 uses a new format for xctest (version 3), which doesn't have a TestSummary.plist file, and uses Zstandard-compressed "database-like" files.

Xcode 11 Release Notes state:

The format of result bundles changed in Xcode 11. A result bundle is an asset produced by Xcode 11 with the xcresult file extension that contains information about the build, tests, code coverage, and more. Any xcresult files produced with Xcode 10 or earlier cannot be read by Xcode 11. A result bundle can be produced by passing -resultBundlePath ./Example.xcresult to an xcodebuild invocation and the Example.xcresult can then be opened in Xcode. Xcode also creates result bundles in Derived Data. The current result bundle version number is 3, which can be specified by passing the xcodebuild flag -resultBundleVersion 3. Version 3 is the default in Xcode 11, but it is still recommended for automation to explicitly pass the flag, so that any potential future versions that become the default do not cause issues to existing tools. Result bundles can be inspected using xcresulttool. A JSON representation of the root object of the result bundle can be exported using xcrun xcresulttool get --format json --path ./Example.xcresult and any nested object, identified by its reference found in the JSON output, can be exported by adding the flag --id REF. xcresulttool also provides the description of its format using xcrun xcresulttool formatDescription. (41633595)

Basically, to obtain results from an Xcode 11 xcresult, one would have to do the following:

// The output of this command will be useful to automatically parse the output of the next commands
$ xcrun xcresulttool formatDescription get --format json
[...]
$ xcrun xcresulttool get --format json --path Project.xcresult
[...]
          "testsRef" : {
            "_type" : {
              "_name" : "Reference"
            },
            "id" : {
              "_type" : {
                "_name" : "String"
              },
              "_value" : "0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=="
            },
[...]
// We take the value of the `testsRef` key (if any) for each `ActionRecord` entry
$ xcrun xcresulttool get --format json --path Project.xcresult --id '0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=='
{
  "_type" : {
    "_name" : "ActionTestPlanRunSummaries"
  },
  "summaries" : {
    "_type" : {
      "_name" : "Array"
    },
    "_values" : [
[...]
snatchev commented 5 years ago

hi @thibault-ml thanks for bringing this up. I will take a look at this once I get some of these JSON files.

thibault-ml commented 5 years ago

Thanks!

Google already parses them: https://chromium.googlesource.com/chromium/src/+/refs/heads/master/ios/build/bots/scripts/ https://github.com/chromium/chromium/tree/master/ios/build/bots/scripts

Bitrise too: https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io/tree/79d467e68e077eddf8f67277a212e42d0b4cd52a/test/converters/xcresult3

And XCResultKit too: https://github.com/davidahouse/XCResultKit

joshdholtz commented 5 years ago

@thibault-ml Looking into! Will hopefully have something for you soon 😊

joshdholtz commented 5 years ago

@thibault-ml Hey! Do you happen to have a xcresult file you could send me? 😊 It would be good to have a legit file that I can test on 🙏

thibault-ml commented 5 years ago

@joshdholtz I'll have a look if we can, not sure 100% I'm at liberty to just yet. Will have a look :-)

joshdholtz commented 5 years ago

@thibault-ml If you don't want to send but are able to run my PR that would also be 💯 😊 The PR works for me on my simple test use cases (I don't have any huge test suites). I got a few fixes to do on it and then will merge.

juliangrosshauser commented 5 years ago

@joshdholtz Here's a .xcresult directory of one of our PSPDFKit test suites. Should contain over 1000 test results:

PSPDFKit.xcresult.zip

Your PR fails with:

bundler: failed to load command: trainer (/Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer)
NoMethodError: [!] undefined method `[]' for nil:NilClass
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:308:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:337:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:337:in `block in initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:336:in `map'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:336:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:238:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:238:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:209:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:209:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:181:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:181:in `block in initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:180:in `map'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:180:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:141:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:141:in `parse_xcresult'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:61:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:47:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:47:in `block in auto_convert'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:33:in `each'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:33:in `auto_convert'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:34:in `block (2 levels) in run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/fastlane-2.130.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:40:in `run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:10:in `start'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/bin/trainer:6:in `<top (required)>'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer:23:in `load'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer:23:in `<top (required)>'
joshdholtz commented 5 years ago

@juliangrosshauser Thanks for sending that over! I had a feeling my PR would need some tweaking for something a little more complex 🙃 I will make sure it works with you sent over ❤️

joshdholtz commented 5 years ago

@juliangrosshauser Got it solved! Thanks sending this 🙌

joshdholtz commented 5 years ago

@juliangrosshauser Alright, PR should be good to go again if you'd like to test when you get some time. I was able to get a successful output from the xcresults that you sent me 👌 I made sure all the data is getting pulled out safely now so that shouldn't happen anymore 😇

juliangrosshauser commented 5 years ago

@joshdholtz Great work! Thank you so much, really appreciate it!

I tested your PR and it works flawlessly now. The only problematic .xcresult directory I could find it this one: PSPDFKit-1-test-failure.zip

It contains 1008 test results. 1007 tests succeed, but the testStyledStringToPDFAsFileURL test fails. The problem is that the resulting XML file contains 1008 successful tests, but no failures.

joshdholtz commented 5 years ago

Oh boy 😱 I found the reason. The testCaseName name in the xcresulttool JSON output is something not as straight forward to match on 🤔 Will work on a solution though

"producingTarget" : {
    "_type" : {
        "_name" : "String"
    },
    "_value" : "PSPDFKitTests"
},
"testCaseName" : {
    "_type" : {
        "_name" : "String"
    },
    "_value" : "-[PSPDFProcessorTests testStyledStringToPDFAsFileURL]"
}
joshdholtz commented 5 years ago

@juliangrosshauser Thanks for testing again! Here is a commit for fixing that 😊 https://github.com/xcpretty/trainer/pull/34/commits/3f88286e27edb2c798ef823042896ea2b4d147b3

juliangrosshauser commented 5 years ago

@joshdholtz I tested your newest changes with Objective-C and Swift failures. Looks like everything works now! :tada: Thank you again!

joshdholtz commented 5 years ago

@juliangrosshauser Thanks so much for feeding me files needed to make this change 😊 I will merge the PR and get a new release out today at some point 💪🚀

vrutberg commented 4 years ago

Thank you guys for working on this!

Is the correct way to use this to specify the -resultBundlePath to xcodebuild, like this, when invoking scan?

xcargs: "-resultBundlePath ./TestSummaries.xcresult"

joshdholtz commented 4 years ago

@vrutberg I think that should work! But you should also be able to do...

scan(
  result_bundle: true,
  output_directory: "./" # optional 
)