Collating test_result bundles are deleted after completion with Xcode 10.3

moatazeldebsy commented 5 years ago

New Issue Checklist

[ ] Updated fastlane-plugin-test_center to the latest version (3.8.7)
[ ] I read the README.md
[ ] I reviewed the example(s) for the action(s) I am using
[ ] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

Issue Description

The parallelization of multi_scan is deleting test_result bundles when it cleans up. These result bundles are stored in the intermediate batch directories for each simultaneous run of tests.

When the test starts it generates 6 different folders (running 6 parallel tests). However, when the test finished it cleans all 6 of these bundles (folders). Screenshot 2019-10-18 at 12 08 28

If we add option `collate_reports:false` the folders not cleaned but we have folders also for re-runs Screenshot 2019-10-18 at 11 51 17

And we have only 23 test cases but without collate, multiscan calculate the re-run tests Screenshot 2019-10-18 at 11 49 54

Environment

Xcode 10.3
Fastlane 2.134.0
Ruby 2.6+
fastlane-plugin-test_center 3.8.7

lyndsey-ferguson commented 5 years ago

Hello @moatazeldebsy, thank you for the bug report. A couple things I want to clarify with you.

multi_scan by default, will collate test_result bundles when given the option to generate them. Collation means that all the test_result bundles are merged into one final test_result bundle. This means that the others are removed.

So, it is a feature to remove the intermediate test_result bundles. Those are intermediate bundles. This is intended behavior.

lyndsey-ferguson commented 5 years ago

When you provide the collate_reports:false option, the test_result bundles are not collated and the intermediate files are not removed because they are not merged into the final test result.

lyndsey-ferguson commented 5 years ago

Unless I misunderstand, this is not an actual bug unless you are asking for a new feature.

moatazeldebsy commented 5 years ago

@lyndsey-ferguson Thank you for your replay and actually the problem that after multi_scan is collating the results from different folders and removed them, the last folder is including only the test output from last clone (in our case clone 6).

Xpretty html report contains the correct number of tests with the correct status but

we are using Allure-reportto generate reports with screenshots and graphs after multi_scan is finished we run this command allure generate fastlane/test_output/UISmokeTests.test_result and allure generate the report for us with these results:

Parse file fastlane/test_output/UISmokeTests.test_result/TestSummaries.plist
Parse file fastlane/test_output/UISmokeTests.test_result/Info.plist
Report successfully generated to allure-report

Then allure open allure-report and we found only 4 tests from 23 tests, and after we check that we found that the last clone number 6 was including 4 tests and this is our problems currently.

lyndsey-ferguson commented 5 years ago

Can you include a log of your fastlane run (please use the --verbose flag)?

moatazeldebsy commented 5 years ago

this is for collate report part

DEBUG [2019-10-18 22:38:24.66]: TestCenter::Helper::HtmlTestReport::Report.collate_report to report:
    <html>
  <head>
    <meta charset='utf-8'/>
    <title>Test Results | xcpretty</title>
    <style type='text/css'>
      body { font-family:Avenir Next, Helvetica Neue, sans-serif; color: #4A4A4A; background-color: #F0F3FB; margin:0;}
      h1 { font-weight: normal; font-size: 24px; margin: 10px 0 0 0;}
      h3 { font-weight: normal; margin: 2px; font-size: 1.1em;}
      header { position: fixed;width: 100%;background: rgba(249, 254, 255, 0.9);margin: 0;padding: 10px;}
      header:before, header:after { content:""; display:table;}
      header:after { clear:both;}
      a:link { color: #A1D761;}
      footer { clear: both;position: relative;z-index: 10;height: 40px;margin-top: -10px; margin-left:30px; font-size:12px;}
      table { width:100%; border-collapse: collapse;}
      tr td:first-child { width:7%}
      .left { float: left; margin-left:30px;}
      .right { float: right; margin-right: 40px; margin-top: 0; margin-bottom:0;}
      .test-suite { margin: 0 0 30px 0;}
      .test-suite > .heading { font-family:Menlo, Monaco, monospace; font-weight: bold; border-color: #A1D761; background-color: #B8E986; border-width: 1px;}
      .test-suite.failing > .heading { border-color: #C84F5E; background-color: #E58591;}
      .test-suite > .heading > .title { margin-top: 4px; margin-left: 10px;}
      .tests { overflow: scroll;margin: 0 30px 0 60px;}
      .test, .test-suite > .heading { height: 30px; overflow: hidden; margin: 0 30px;}
      .test, .test-suite > .heading { border-width: 1px; border-collapse: collapse; border-style: solid; }
      .test { margin-left: 30px; border-top:none;}
      .test.failing { border-color: #C84F5E; background-color: #F4DDE0;}
      .test.passing { border-color: #A1D761;}
      .test.failing { background-color: #E7A1AA;}
      .test.passing { background-color: #CAF59F;}
      .test.failing.odd { background-color: #EEC7CC;}
      .test.passing.odd { background-color: #E5FBCF;}
      .details.failing { background-color: #F4DDE0; border: 1px solid #C84F5E;}
      .details.passing { background-color: #E5F4DC; border: 1px solid #A1D761;}
      .test .test-detail:last-child { padding-bottom: 8px;}
      .test .title { float: left; font-size: 0.9em; margin-top: 8px; font-family: Menlo, Monaco, monospace;}
      .test .time { float: left;margin: 4px 10px 0 20px;}
      .test-detail { font-family:Menlo, Monaco, monospace; font-size: 0.9em; margin: 5px 0 5px 0px;}
      .screenshots { height: auto; overflow: hidden; padding: 4px 4px 0 4px; background-color: #B8E986; border: #A1D761; border-width: 0 1px; border-style: solid; }
      .screenshots.failing { border-color: #C84F5E; background-color: #E58591; }
      .screenshot { max-height: 60px; float: left; transition: max-height 0.2s; margin: 0 4px 4px 0 }
      .screenshot.selected { max-height: 568px; }
      #test-suites { display: inline-block; width: 100%;margin-top:100px;}
      #segment-bar { margin-top: 10px;margin-left: 14px;float:right;}
      #segment-bar a:first-child { border-radius: 9px 0 0 9px; border-right: none;}
      #segment-bar a:last-child { border-radius: 0 9px 9px 0; border-left: none;}
      #segment-bar > a { color: #565656; border: 2px solid  #7B7B7B; width: 80px; font-weight: bold; display:inline-block;text-align:center; font-weight: normal;}
      #segment-bar > a.selected { background-color: #979797; color: #F0F3FB;}
      #counters { float: left;margin: 10px;text-align: right;}
      #counters h2 { font-size: 16px; font-family: Avenir, sans-serif; font-weight: lighter; display:inline;}
      #counters .number { font-size: 20px;}
      #fail-count { color: #D0021B; margin-left:10px;}
      @media (max-width: 640px) {
        h1, #counters, #segment-bar { margin: 5px auto; text-align:center;}
        header, #segment-bar { width: 100%; position: relative; background:none;}
        .left, .right { float:none; margin:0;}
        #test-suites { margin-top: 0;}
        #counters { float:none;}
      }
    </style>
    <script type='text/javascript'>
      var hide = function(element) { element.style.display = 'none';}
      var show = function(element) { element.style.display = '';}
      var isHidden = function(element) { return element.style.display == 'none';}
      var isSelected = function(element) { return element.classList.contains("selected");}
      var deselect = function(element) { return element.classList.remove("selected");}
      var select = function(element) { return element.classList.add("selected");}
      var toggle = function(element) { isHidden(element) ? show(element) : hide(element);};
      var toggleTests = function(heading) { toggle(heading.parentNode.children[1]);};
      var toggleDetails = function(detailClass) {
        var details = document.querySelectorAll('.' + detailClass);
        for (var i = details.length - 1; i >= 0; i--) { toggle(details[i]);};
      };
      var hideAll = function(collection) {
        for (var i = collection.length - 1; i >= 0; i--) { hide(collection[i]); };
      }
      var showAll = function(collection) {
        for (var i = collection.length - 1; i >= 0; i--) { show(collection[i]); };
      }
      var selectSegment = function(segment) {
        if (isSelected(segment)) return;
        var segments = document.querySelectorAll('#segment-bar > a');
        for (var i = segments.length - 1; i >= 0; i--) { deselect(segments[i]);};
        select(segment);
        if (segment.id == "all-segment") {
          showAll(document.querySelectorAll('.test-suite'));
          showAll(document.querySelectorAll('.test'));
        } else if (segment.id == "failing-segment") {
          hideAll(document.querySelectorAll('.test.passing'));
          showAll(document.querySelectorAll('.test.failing'));
          hideAll(document.querySelectorAll('.test-suite.passing'));
          showAll(document.querySelectorAll('.test-suite.failing'));
        } else if (segment.id == "passing-segment") {
          hideAll(document.querySelectorAll('.test.failing'));
          showAll(document.querySelectorAll('.test.passing'));
          hideAll(document.querySelectorAll('.test-suite.failing'));
          showAll(document.querySelectorAll('.test-suite.passing'));
        }
      }
      var toggleScreenshot = function(suiteName, index) {
        var screenshot = document.getElementById("screenshot-" + suiteName + "-" + index);
        isSelected(screenshot) ? deselect(screenshot) : select(screenshot);
      }
    </script>
  </head>
  <body>
    <header>
      <section class='left'>
        <h1>Test Results</h1>
      </section>
      <section class='right'>
        <section id='counters'>
          <h2 id='test-count'><span class='number'>23</span> tests</h2>

        <h2 id='fail-count'><span class='number'>3</span></h2></section>
        <section id='segment-bar'>
          <a class='selected' id='all-segment' onclick='selectSegment(this);'>All</a><a id='failing-segment' onclick='selectSegment(this);'>Failing</a><a id='passing-segment' onclick='selectSegment(this);'>Passing</a>
        </section>
      </section>
    </header>
    <section id='test-suites'>

        <section class='test-suite passing' id='VoloUITests.CartCalculationUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.CartCalculationUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testCartSubtotalCalculation&apos;);'>
                <td>

                    <h3 class='time'>62.072s</h3>

                </td>
                <td><h3 class='title'>testCartSubtotalCalculation</h3></td>
              </tr>

            </table>
          </section>
        </section>

        <section class='test-suite passing' id='VoloUITests.ProductsManipulationUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.ProductsManipulationUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testDecreaseProductQuantity&apos;);'>
                <td>

                    <h3 class='time'>48.792s</h3>

                </td>
                <td><h3 class='title'>testDecreaseProductQuantity</h3></td>
              </tr>

              <tr class='test passing  odd' onclick='toggleDetails(&apos;testIncreaseProductQuantity&apos;);'>
                <td>

                    <h3 class='time'>53.714s</h3>

                </td>
                <td><h3 class='title'>testIncreaseProductQuantity</h3></td>
              </tr>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testRemoveProduct&apos;);'>
                <td>

                    <h3 class='time'>47.312s</h3>

                </td>
                <td><h3 class='title'>testRemoveProduct</h3></td>
              </tr>

              <tr class='test passing  odd' onclick='toggleDetails(&apos;testRemoveProductWhenCountGoesToZero&apos;);'>
                <td>

                    <h3 class='time'>45.467s</h3>

                </td>
                <td><h3 class='title'>testRemoveProductWhenCountGoesToZero</h3></td>
              </tr>

            </table>
          </section>
        </section>

    <section class='test-suite passing' id='VoloUITests.ProfileEditingUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.ProfileEditingUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testChangeMobileNumber&apos;);'>
                <td>

                    <h3 class='time'>80.475s</h3>

                </td>
                <td><h3 class='title'>testChangeMobileNumber</h3></td>
              </tr>

              <tr class='test passing odd' onclick='toggleDetails(&apos;testChangePassword&apos;);'>
                <td>

                    <h3 class='time'>102.391s</h3>

                </td>
                <td><h3 class='title'>testChangePassword</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing' id='VoloUITests.RestaurantMenuUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.RestaurantMenuUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testAddProductsToCart&apos;);'>
                <td>

                    <h3 class='time'>51.064s</h3>

                </td>
                <td><h3 class='title'>testAddProductsToCart</h3></td>
              </tr>

              <tr class='test passing odd' onclick='toggleDetails(&apos;testAddProductsWithVariationsToCart&apos;);'>
                <td>

                    <h3 class='time'>53.810s</h3>

                </td>
                <td><h3 class='title'>testAddProductsWithVariationsToCart</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing' id='VoloUITests.PaypalHPPUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.PaypalHPPUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testNewPayment&apos;);'>
                <td>

                    <h3 class='time'>21.040s</h3>

                </td>
                <td><h3 class='title'>testNewPayment</h3></td>
              </tr>

              <tr class='test passing  odd' onclick='toggleDetails(&apos;testTokenizedPayment&apos;);'>
                <td>

                    <h3 class='time'>11.127s</h3>

                </td>
                <td><h3 class='title'>testTokenizedPayment</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing' id='VoloUITests.VoucherUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.VoucherUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testCalculationWithAmountVoucher&apos;);'>
                <td>

                    <h3 class='time'>104.826s</h3>

                </td>
                <td><h3 class='title'>testCalculationWithAmountVoucher</h3></td>
              </tr>

              <tr class='test passing odd' onclick='toggleDetails(&apos;testCalculationWithFreeDeliveryVoucher&apos;);'>
                <td>

                    <h3 class='time'>88.721s</h3>

                </td>
                <td><h3 class='title'>testCalculationWithFreeDeliveryVoucher</h3></td>
              </tr>

              <tr class='test passing ' onclick='toggleDetails(&apos;testCalculationWithPercentageVoucher&apos;);'>
                <td>

                    <h3 class='time'>81.618s</h3>

                </td>
                <td><h3 class='title'>testCalculationWithPercentageVoucher</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing ' id='VoloUITests.PaypalSDKUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.PaypalSDKUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testNewUser&apos;);'>
                <td>

                    <h3 class='time'>117.503s</h3>

                </td>
                <td><h3 class='title'>testNewUser</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing ' id='VoloUITests.RegistrationUITest'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.RegistrationUITest</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testUserWithEmail&apos;);'>
                <td>

                    <h3 class='time'>40.223s</h3>

                </td>
                <td><h3 class='title'>testUserWithEmail</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing' id='VoloUITests.AddressUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.AddressUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;test01UserCanSearchAddressAndGetRestaurants&apos;);'>
                <td>

                    <h3 class='time'>65.152s</h3>

                </td>
                <td><h3 class='title'>test01UserCanSearchAddressAndGetRestaurants</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing' id='VoloUITests.LoginUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.LoginUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testUserWithEmail&apos;);'>
                <td>

                    <h3 class='time'>56.840s</h3>

                </td>
                <td><h3 class='title'>testUserWithEmail</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite passing' id='VoloUITests.ReorderUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.ReorderUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testQuickReorderOrderPlaced&apos;);'>
                <td>

                    <h3 class='time'>63.615s</h3>

                </td>
                <td><h3 class='title'>testQuickReorderOrderPlaced</h3></td>
              </tr>

            </table>
          </section>
        </section><section class='test-suite failing' id='VoloUITests.AddressBookUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.AddressBookUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test failing ' onclick='toggleDetails(&apos;testUserCanAddNewAddressWithEditAndDeletion&apos;);'>
                <td>

                </td>
                <td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
              </tr>

                <tr class='details failing testUserCanAddNewAddressWithEditAndDeletion'>
                  <td/>
                  <td>

                      <section class='test-detail reason'>XCTAssertTrue failed</section>

                      <section class='test-detail snippet'/>
                      <section class='test-detail'>AddressBookUITests.swift:45</section>

                  </td>
                </tr>

            </table>
          </section>
        </section><section class='test-suite failing' id='VoloUITests.CashOnDeliveryUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.CashOnDeliveryUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test failing ' onclick='toggleDetails(&apos;testNewUserWithEmail&apos;);'>
                <td>

                </td>
                <td><h3 class='title'>testNewUserWithEmail</h3></td>
              </tr>

                <tr class='details failing testNewUserWithEmail'>
                  <td/>
                  <td>

                      <section class='test-detail reason'>No matches found for Find: Elements matching predicate '"closeXButton" IN identifiers' from input {(</section>

                      <section class='test-detail snippet'/>
<section class='test-detail'>&lt;unknown&gt;:0</section>

                  </td>
                </tr>

              <tr class='test failing odd' onclick='toggleDetails(&apos;testReturningUserEmail&apos;);'>
                <td>

                </td>
                <td><h3 class='title'>testReturningUserEmail</h3></td>
              </tr>

                <tr class='details failing testReturningUserEmail'>
                  <td/>
                  <td>

                      <section class='test-detail reason'>XCTAssertTrue failed</section>

                      <section class='test-detail snippet'/>
                      <section class='test-detail'>CashOnDeliveryUITests.swift:33</section>

                  </td>
                </tr>

              <tr class='test passing ' onclick='toggleDetails(&apos;testReturningUserFacebook&apos;);'>
                <td>

                    <h3 class='time'>77.157s</h3>

                </td>
                <td><h3 class='title'>testReturningUserFacebook</h3></td>
              </tr>

            </table>
          </section>
        </section></section>
    <footer>Report generated with <a href='https://github.com/supermarin/xcpretty'>xcpretty</a></footer>
  </body>
</html>
DEBUG [2019-10-18 22:38:24.69]:     collating testsuite
        <section class='test-suite passing' id='VoloUITests.AddressBookUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.AddressBookUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testUserCanAddNewAddressWithEditAndDeletion&apos;);'>
                <td>

                    <h3 class='time'>79.024s</h3>

                </td>
                <td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
              </tr>

            </table>
          </section>
        </section>
DEBUG [2019-10-18 22:38:24.69]:         updating testcase
            <tr class='test failing ' onclick='toggleDetails(&apos;testUserCanAddNewAddressWithEditAndDeletion&apos;);'>
                <td>

                </td>
                <td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
              </tr>
DEBUG [2019-10-18 22:38:24.71]:     after collation exiting testsuite
        <section class='test-suite passing ' id='VoloUITests.AddressBookUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.AddressBookUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testUserCanAddNewAddressWithEditAndDeletion&apos;);'>
                <td>

                    <h3 class='time'>79.024s</h3>

                </td>
                <td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
              </tr>

            </table>
          </section>
        </section>
DEBUG [2019-10-18 22:38:24.73]:     collating testsuite
        <section class='test-suite passing' id='VoloUITests.CashOnDeliveryUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.CashOnDeliveryUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing ' onclick='toggleDetails(&apos;testNewUserWithEmail&apos;);'>
                <td>

                    <h3 class='time'>88.714s</h3>

                </td>
                <td><h3 class='title'>testNewUserWithEmail</h3></td>
              </tr>

              <tr class='test passing odd' onclick='toggleDetails(&apos;testReturningUserEmail&apos;);'>
                <td>

                    <h3 class='time'>60.746s</h3>

                </td>
                <td><h3 class='title'>testReturningUserEmail</h3></td>
              </tr>

              <tr class='test passing ' onclick='toggleDetails(&apos;testReturningUserFacebook&apos;);'>
                <td>

                    <h3 class='time'>65.932s</h3>

                </td>
                <td><h3 class='title'>testReturningUserFacebook</h3></td>
              </tr>

            </table>
          </section>
        </section>
DEBUG [2019-10-18 22:38:24.77]:         updating testcase
            <tr class='test failing ' onclick='toggleDetails(&apos;testNewUserWithEmail&apos;);'>
                <td>

                </td>
                <td><h3 class='title'>testNewUserWithEmail</h3></td>
              </tr>
DEBUG [2019-10-18 22:38:24.78]:         updating testcase
            <tr class='test failing odd' onclick='toggleDetails(&apos;testReturningUserEmail&apos;);'>
                <td>

                </td>
                <td><h3 class='title'>testReturningUserEmail</h3></td>
              </tr>
DEBUG [2019-10-18 22:38:24.79]:         updating testcase
            <tr class='test passing ' onclick='toggleDetails(&apos;testReturningUserFacebook&apos;);'>
                <td>

                    <h3 class='time'>77.157s</h3>

                </td>
                <td><h3 class='title'>testReturningUserFacebook</h3></td>
              </tr>
DEBUG [2019-10-18 22:38:24.80]:     after collation exiting testsuite
        <section class='test-suite passing ' id='VoloUITests.CashOnDeliveryUITests'>
          <section class='heading' onclick='toggleTests(this);'>
            <h3 class='title'>VoloUITests.CashOnDeliveryUITests</h3>
          </section>
          <section class='tests'>
            <table>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testNewUserWithEmail&apos;);'>
                <td>

                    <h3 class='time'>88.714s</h3>

                </td>
                <td><h3 class='title'>testNewUserWithEmail</h3></td>
              </tr>

              <tr class='test passing  odd' onclick='toggleDetails(&apos;testReturningUserEmail&apos;);'>
                <td>

                    <h3 class='time'>60.746s</h3>

                </td>
                <td><h3 class='title'>testReturningUserEmail</h3></td>
              </tr>

              <tr class='test passing  ' onclick='toggleDetails(&apos;testReturningUserFacebook&apos;);'>
                <td>

                    <h3 class='time'>65.932s</h3>

                </td>
                <td><h3 class='title'>testReturningUserFacebook</h3></td>
              </tr>

            </table>
          </section>
        </section>
DEBUG [2019-10-18 22:38:24.99]: Collating test_result bundles ["/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-3/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-2/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-5/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-6/FoodpandaUISmokeTests-2.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-4/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-1/FoodpandaUISmokeTests-2.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-6/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-1/FoodpandaUISmokeTests.test_result"]
DEBUG [2019-10-18 22:38:25.05]: result bundles are NOT of format version 3
INFO [2019-10-18 22:39:21.91]: Finished collating test_result bundle to '/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests/FoodpandaUISmokeTests.test_result'
DEBUG [2019-10-18 22:39:30.49]: Results for each test run: [false, true, true, true, true, false]

+-------------------+-----------------------------------------------------------------------+
|                                    multi_scan results                                     |
+-------------------+-----------------------------------------------------------------------+
| result            | false                                                                 |
| total_tests       | 23                                                                    |
| passing_testcount | 23                                                                    |
| failed_testcount  | 0                                                                     |
| failed_tests      | []                                                                    |
| total_retry_count | 8                                                                     |
| report_files      | ["/fastlane/test_output/report.junit"  |
|                   | ,                                                                     |
|                   | "/fastlane/test_output/report.html",   |
|                   | "/pd-mob-b2c-ios/fastlane/test_output/FoodpandaUISmo  |
|                   | keTests.test_result"]                                                 |
+-------------------+-----------------------------------------------------------------------+

INFO [2019-10-18 22:39:30.61]: Cruising back to lane 'ios ta_foodpanda_smoke' 🚘
INFO [2019-10-18 22:39:30.61]: -------------------
INFO [2019-10-18 22:39:30.61]: --- Step: slack ---
INFO [2019-10-18 22:39:30.61]: -------------------
INFO [2019-10-18 22:39:31.28]: Successfully sent Slack notification
INFO [2019-10-18 22:39:31.29]: Successfully generated documentation at path '/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/README.md'

+------+-----------------------------------------+-------------+
|                       fastlane summary                       |
+------+-----------------------------------------+-------------+
| Step | Action                                  | Time (in s) |
+------+-----------------------------------------+-------------+
| 1    | Verifying fastlane version              | 0           |
| 2    | default_platform                        | 0           |
| 3    | Switch to ios build_run_ta_testing lane | 0           |
| 4    | multi_scan                              | 886         |
| 5    | slack                                   | 0           |
+------+-----------------------------------------+-------------+

DEBUG [2019-10-18 22:39:31.30]: All plugins are up to date
INFO [2019-10-18 22:39:31.31]: fastlane.tools just saved you 15 minutes! 🎉
DEBUG [2019-10-18 22:39:31.31]: All plugins are up to date
INFO [2019-10-18 22:39:31.31]: Deleting iPhone Xʀ Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:39:38.78]: Deleting iPhone Xʀ Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:39:46.04]: Deleting iPhone Xʀ Clone 3 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:39:53.07]: Deleting iPhone Xʀ Clone 4 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:40:00.02]: Deleting iPhone Xʀ Clone 5 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:40:06.97]: Deleting iPhone Xʀ Clone 6 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>

lyndsey-ferguson commented 5 years ago

Ah, okay. Yes, collate_test_result_bundles will not properly collate pre-version 3 format bundles. With Xcode 11, it will. It is a lot of work to collate pre-Xcode 11 based test_results properly and Xcode 11 does it with one command, so I have decided to not do the work.

Please update to Xcode 11 and this will begin working.

lyndsey-ferguson commented 5 years ago

I will close this issue as I don't intend on fixing it. If you feel strongly about it and want to fix that action to work pre-Xcode-11, I can give you guidance on making the changes.

lyndsey-ferguson commented 5 years ago

@moatazeldebsy just think, if you do fix this for Xcode 10 people, you'll become famous in the test_center community!! You'll be the unsung hero!

Let me know if you'd like to tackle this. It is not complicated work, just a series of things that you need to do that take some time. Time I don't have to do it by myself.

lyndsey-ferguson commented 5 years ago

I would review this PR where I fixed collation for HTML test results. I would suggest the same approach. Make wrapper classes for each file of a Xcode 10 test result with methods to "collate" each file.

moatazeldebsy commented 5 years ago

@lyndsey-ferguson ok will try to do that. I need to check this PR or need to do something else

moatazeldebsy commented 5 years ago

@lyndsey-ferguson I updated to Xcode 11 and the results are in xcresult format without screenshots and logs.

inside test_results

inside scheme folder

insdie data folder

DEBUG [2019-10-19 15:17:51.59]:     adding testsuite
        #<TestCenter::Helper::HtmlTestReport::TestSuite:0x00007fa932ee9c50>
DEBUG [2019-10-19 15:17:51.61]:     adding testsuite
        #<TestCenter::Helper::HtmlTestReport::TestSuite:0x00007fa932ee9bb0>
DEBUG [2019-10-19 15:17:51.74]: Collating test_result bundles ["/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-3/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-2/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-5/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-4/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-6/FoodpandaUISmokeTests-4.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-1/FoodpandaUISmokeTests-2.test_result"]
DEBUG [2019-10-19 15:17:51.74]: result bundles are of format version 3
INFO [2019-10-19 15:17:51.81]: xcrun xcresulttool merge /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1kblznm.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-14t8sg5.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1yjpx97.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1nkujtl.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-4.test_result20191019-99406-1dvadtf.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-2.test_result20191019-99406-17356zd.xcresult --output-path /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/d20191019-99406-1itt0gc/FoodpandaUISmokeTests.test_result
INFO [2019-10-19 15:17:51.81]: $ xcrun xcresulttool merge /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1kblznm.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-14t8sg5.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1yjpx97.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1nkujtl.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-4.test_result20191019-99406-1dvadtf.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-2.test_result20191019-99406-17356zd.xcresult --output-path /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/d20191019-99406-1itt0gc/FoodpandaUISmokeTests.test_result
INFO [2019-10-19 15:17:59.14]: ▸ [v3] Merged to: /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/d20191019-99406-1itt0gc/FoodpandaUISmokeTests.test_result
INFO [2019-10-19 15:18:00.49]: Finished collating test_result bundle to '/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests/FoodpandaUISmokeTests.test_result'
DEBUG [2019-10-19 15:18:02.60]: Results for each test run: [true, true, true, true, true, false]

Environment

Xcode 11.1
Fastlane 2.134.0
Ruby 2.6+
fastlane-plugin-test_center (3.8.10)

with fastlane-plugin-test_center (3.8.7) all folders for batches already exists and including xcresult only

lyndsey-ferguson commented 5 years ago

@moatazeldebsy are you saying that the original test result bundles have screenshots and then when the xcresulttool performs the merge, the screenshots are no longer there? Can you please post sample test result bundles that demonstrate the failing merge?

(My Xcode 11 install is broken and I cannot get tests to run on my local computer at the moment.)

moatazeldebsy commented 5 years ago

Yes during running the tests the folders have screenshots and logs but in the collate step (last step) all removed and I found only the xcresult files.

And this is the final result for each batch and folders exist for all batches without logs or screenshots

we have 1 folder including index and JUnit report

lyndsey-ferguson commented 5 years ago

Ok. Please post sample test result bundles so that I can test that out. Once you do that, I'll update this Issue and reopen it.

moatazeldebsy commented 5 years ago

sample test result bundles from the logs or the folders?, I didn't get it

lyndsey-ferguson commented 5 years ago

I need sample result bundles. If you could provide your call the multi_scan the option collate_reports: false, it will generate uniquely named test result bundles. Please zip those up and and attach them to this Issue (make sure no sensitive data is provided).

jineshqa commented 5 years ago

@lyndsey-ferguson I would like to add that .xctestresult files are altogether missing. I use xchtmlreport to generate xcode type looking result html. This plugin will soon be upgraded to support Xcode 11 and would need .xcresultfiles to generate html. Could you not delete .xcresult file or collate 1 .xcresult file from multiple simulator runs?

lyndsey-ferguson commented 5 years ago

From my observation, the xcresult file is just a test_result bundle with the xcresult file extension.

jineshqa commented 5 years ago

So far the new version of the plugin is not released. As soon as it is released, I'll test it out and let you know. In worst case could we get a collated .xcresult file instead of bundle?

lyndsey-ferguson commented 5 years ago

You may have to make a symbolic link ending with .xcresult that points to your resulting test_result bundle and provide that the the xchtmlreport plugin, but I really don't know. Play with it and let me know 😄

moatazeldebsy commented 5 years ago

@lyndsey-ferguson test_results is big file but like you said test_result bundle with the xcresult file extension without any screenshot or logs even I disable collate this is the results from 1 clone like the following image

lyndsey-ferguson commented 5 years ago

@moatazeldebsy can you upload the bundles to a service like Dropbox? I need to run the code to see where it is failing and what is in the intermediate test_result bundles.

moatazeldebsy commented 4 years ago

Hi @lyndsey-ferguson can I know with which Xcode version you fixed this issue Fixes issue with multi_scan when collating test_result?

lyndsey-ferguson commented 4 years ago

Sorry, I don't remember.

lyndsey-ferguson commented 4 years ago

I believe I only started using Xcode 11 around Oct 18th.

lyndsey-ferguson / fastlane-plugin-test_center