Detailed Test Report - Missing?

[TedHusted]

I've made my first successful run of ApexUnit, and I'm eager to make it part of our continuous delivery and continuous support processes. Thank you so much.

The GitHub page says

---

On successful completion - the command should give you build success and it should generate two report files - ApexUnitReport.xml (This is the test report in JUnit format) and Report/ApexUnitReport.html (This is the code coverage report in html format)

---

And I see both of those files (attached). The ApexUnitReport.html includes this link:

---

<a style="font-size:125%"; href=/opt/ApexUnit/ApexUnitReport.xml>Detailed Test Report

---

Is this link working as designed?

My expectation would be that link would lead to a ApexUnitReportDetail.html or some such.

A hard-coded link to the original XML file doesn't seem correct.

Although the files are generated, I am seeing a stack trace.

---

[com.sforce.cd.apexUnit.ApexUnitRunner.main()] INFO com.sforce.cd.apexUnit.ApexUnitUtils - Shutting down ApexUnit java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1365) at com.sforce.cd.apexUnit.ApexUnitUtils.shutDownWithErrMsg(ApexUnitUtils.java:68) at com.sforce.cd.apexUnit.ApexUnitRunner.main(ApexUnitRunner.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297) at java.lang.Thread.run(Thread.java:745)

Earlier, there is also a cookie warning

---

WARNING: Cookie rejected: "$Version=0; BrowserId=MkyULpRrSUO0IoBBw13Ejw; $Path=/; $Domain=.salesforce.com". Domain attribute ".salesforce.com" violates RFC 2109: host minus domain may not contain any dots

---

The org uses a my domain, which I specify in the login url.

org.login.url https://nc-source.my.salesforce.com

I'm using a fresh checkout of ApexUnit and Maven 3.3.9 on Java 1.7.0_95

Is there suppose to be a detail page available?

Is the error preventing the page from being completed?

ApexUnitReport.xml.txt

ApexUnitReport.html.txt

[adarsh-ramakrishna-sfdc]

Hi @TedHusted Congrats on your successful ApexUnit run!

1. Detailed Test Report link points to the ApexUnitReport.xml. You would find this in the file: https://github.com/forcedotcom/ApexUnit/blob/master/src/main/java/com/sforce/cd/apexUnit/report/ApexCodeCoverageReportGenerator.java apexUnitTestReportPath = System.getProperty("user.dir") + System.getProperty("file.separator") + "ApexUnitReport.xml"; The intent of the link is: if a CI engine like jenkins is used, you get to navigate to the test report from the code coverage(html) report using this link.. This link is placed right below test execution summary so if a user wants to take a look at the test report, the user is just a click away! We don't have more details to share in ApexUnitReportDetail.html ; all the details are provided in ApexUnitReport.html. Let us know your thoughts.

1. Do you see any error message with your stack trace? Can you please share it?
2. The cookie rejected warning is mentioned in https://github.com/forcedotcom/ApexUnit/issues/28 and its unresolved so far.

[TedHusted]

Thank you for the prompt response.

My system is only showing the two files

---

/ApexUnitReport.xml /Reports/ApexUnitReport.html

---

ls ApexUnitReport.xml pom.xml Report Slide Deck.pdf target license.txt README.md SFDC_CLA.pdf src

ls Report ApexUnitReport.html

---

code coverage(html) report using this link..

I'm not seeing a code coverage report, and I'm not making the connection between linking to "ApexUnitReport.xml" and code coverage(html).

The report on the slide deck seems to be showing quite a bit more that my report shows.

if a CI engine like jenkins is used,

Another use case is to surface the report for people who don't have access to the CI server, meaning that the links should be relative and portable, so the script could deploy the files to a web server.

I'd like to do dozens of these reports for our various customers, and I would need to deploy them to a logical structure on a web server, with a reliable URL.

-Ted.

[TedHusted]

Thank you, Thomas. I appreciate the response.

In your project, what presents when you click the "Detailed Test Report" link?

In my report, the link is hard coded to the XML file (literally, "/opt/ApexUnit/ApexUnitReport.xml"), which doesn't seem correct.

-Ted.

-Ted

On Thu, Mar 3, 2016 at 6:08 AM, Thomas Fuda notifications@github.com wrote:

@TedHusted https://github.com/TedHusted - Please take a look at this screenshot of my Eclipse project for running ApexUnit: http://screencast.com/t/pKR0pObV. You can see that ApexUnitReport.html is created in a Report folder off of the project root folder, whereas the ApexUnitReport.xml file is created directly in the project root.

— Reply to this email directly or view it on GitHub https://github.com/forcedotcom/ApexUnit/issues/32#issuecomment-191710487 .

[tfuda]

In my case, the "Detailed Test Report" link is the absolute file URL for the XML file: file:///C:/Users/tom/Development/Maven/git/ApexUnit/ApexUnitReport.xml. When I click the link, it opens the XML document in the web browser. I don't really think the ApexUnit developers have considered your use case for third parties that need to see your test results without having access to the CI server. I think this is intended more for the CI server to parse the test results locally, allowing it to report failures.

On Thu, Mar 3, 2016 at 8:59 AM, Ted Husted notifications@github.com wrote:

Thank you, Thomas. I appreciate the response.

In your project, what presents when you click the "Detailed Test Report" link?

In my report, the link is hard coded to the XML file (literally, "/opt/ApexUnit/ApexUnitReport.xml"), which doesn't seem correct.

-Ted.

-Ted

On Thu, Mar 3, 2016 at 6:08 AM, Thomas Fuda notifications@github.com wrote:

@TedHusted https://github.com/TedHusted - Please take a look at this screenshot of my Eclipse project for running ApexUnit: http://screencast.com/t/pKR0pObV. You can see that ApexUnitReport.html is created in a Report folder off of the project root folder, whereas the ApexUnitReport.xml file is created directly in the project root.

— Reply to this email directly or view it on GitHub < https://github.com/forcedotcom/ApexUnit/issues/32#issuecomment-191710487> .

— Reply to this email directly or view it on GitHub https://github.com/forcedotcom/ApexUnit/issues/32#issuecomment-191771871 .

Tom Fuda Nine Summer Email: tom@9summer.com Phone: +1.203.361.9416 http://9summer.com : Accelerating innovative products to market.

[TedHusted]

Besides wanting to stand up the reports for dozens of orgs, my other problem is that for one of our critical development orgs, the Apex tests report false negatives if run asynchronously, and so we are compelled to run them synchronously, which takes literally 90 to 120 minutes.

I don't want to tie up one of our TeamCity build agents for that long, so my intention is to run it as a background task, and I can't rely on a build server to process the file.

Is parsing the second file beyond the capabilities of Maven? I don't understand why ApexUnit doesn't render the second report itself. Is that feature on the roadmap?

[tfuda]

I feel your pain. Dave Esposito and I decided to look at ApexUnit for the exact same reason as you. Our builds were taking up to about 50 minutes. Most of that time spent executing 500+ unit tests. We also hit intermittent unit test failures when the unit tests are run asynchronously, due to occasional record locking errors. So, what we did here is fork the project and implement our own retry mechanism. We created a Java "main" class that invokes ApexUnit with appropriate command line args. If after running the full test suite, there are errors, it parses the ApexUnitReport.xml file and feeds the failed tests back into ApexUnit for a second pass. We got it to the point where it is pretty much working the way we want, but we shelved the effort for a bit, because of issue #22. We wanted to make sure we are always above the 75% code coverage numbers required for packaging, however, there's no way to specify a code coverage regex that would exclude our test classes from the code coverage computations. And also, one of our CI orgs is on NA7. For whatever reason, I have been unable to EVER get a successful run of ApexUnit against this org. I documented this in issue #11.

We intend that some day we will submit this as a pull request for enhancement to the ApexUnit project, or perhaps just make it available as a separate GitHub project as an add-on to ApexUnit, but right now it's sort of been mothballed due to the other issues I identified above.

On Thu, Mar 3, 2016 at 9:47 AM, Ted Husted notifications@github.com wrote:

Besides wanting to stand up the reports for dozens of orgs, my other problem is that for one of our critical development orgs, the Apex tests report false negatives if run asynchronously, and so we are compelled to run them synchronously, which takes literally 90 to 120 minutes.

I don't want to tie up one of our TeamCity build agents for that long, so my intention is to run it as a background task, and I can't rely on a build server to process the file.

Is parsing the second file beyond the capabilities of Maven? I don't understand why ApexUnit doesn't render the second report itself. Is that feature on the roadmap?

— Reply to this email directly or view it on GitHub https://github.com/forcedotcom/ApexUnit/issues/32#issuecomment-191796117 .

Tom Fuda Nine Summer Email: tom@9summer.com Phone: +1.203.361.9416 http://9summer.com : Accelerating innovative products to market.

[pmedapuram]

@tfuda and @TedHusted
This feedback is great. We are constantly working towards getting ApexUnit smarter and flexible to address the common problems.

I see that being able to run Apex Test classes asynchronously without any issues, is an important aspect of this tool. We are planning on addressing such issues faced with ApexUnit.

Regarding https://github.com/forcedotcom/ApexUnit/issues/22 , definitely agree with not computing code coverage for test classes. We are looking to get to this and resolve it soon. It is on our road map.

@TedHusted We are looking at your reports issue.

[tfuda]

I doubt that you will ever get the Apex classes to run asychronously, without failure, 100% of the time, (I think its the nature of the platform, and the unit tests themselves, rather than an ApexUnit issue) so I think it's more important to try and work some kind of re-try mechanism in the framework.

On Thu, Mar 3, 2016 at 2:05 PM, Pavan Tej Medapuram < notifications@github.com> wrote:

@tfuda https://github.com/tfuda and @TedHusted https://github.com/TedHusted

This feedback is great. We are constantly working towards getting ApexUnit smarter and flexible to address the common problems.

I see that being able to run Apex Test classes asynchronously without any issues, is an important aspect of this tool. We are planning on addressing such issues faced with ApexUnit.

Regarding #22 https://github.com/forcedotcom/ApexUnit/issues/22 , definitely agree with not computing code coverage for test classes. We are looking to get to this and resolve it soon. It is on our road map.

@TedHusted https://github.com/TedHusted We are looking at your reports issue.

— Reply to this email directly or view it on GitHub https://github.com/forcedotcom/ApexUnit/issues/32#issuecomment-191917224 .

Tom Fuda Nine Summer Email: tom@9summer.com Phone: +1.203.361.9416 http://9summer.com : Accelerating innovative products to market.

[pmedapuram]

Yes @tfuda , We may have to build-in some a mechanism to take care of these failures. Will share the details with you once we have a concrete plan. We could work together on this.

[adarsh-ramakrishna-sfdc]

@tfuda Retry mechanism is on its way as part of the fix for #11 . @TedHusted Yes, currently the "Detailed Test Report" link points to the ApexUnit.xml(JUnit style test report) and considers the absolute path of the file in the project setup. I like the portable link idea..

You did share in the first post that the tool threw error in spite of executing ApexUnit, Can you please share the error logs?

[TedHusted]

@tfuda> feeds the failed tests back into ApexUnit for a second pass

Does this imply that ApexUnit can gather the coverage/outcome from tests that weren't run in the current process.

Could we build/run the tests separately and then run ApexUnit afterwards?

[adarsh-ramakrishna-sfdc]

@TedHusted Can you please re-iterate your request here? Looking at all the posts/comments here, it seems like you were not clear on "Detailed Test Report" link. Now that various participants have provided their answers, do you need anything else from this issue record? Are you looking for a portable link to the test report? Please clarify,

[TedHusted]

Thanks, Adarsh. I appreciate the followup,

The intent of the link is: if a CI engine like jenkins is used, you get to navigate to the test report from the code coverage(html) report using this link.

We don't use Jenkins anymore, but from

• https://github.com/forcedotcom/ApexUnit/wiki/Integrating-with-CI-pipeline

it sounds like we might want a link back to an index.html, but I don't see where a hard-link to ApexUnitReport.xml fits in. The location of the JUnit report file is specified in the configuration.

(1) Add "Publish HTML reports" step and populate 'HTML directory to archive' field with 'Report' and 'Index page[s]' field with 'ApexUnitReport.html' and choose your 'Report title'

(2) Add "Publish JUnit test result report" step and populate 'ApexUnitReport.xml' for the 'Test Report XMLs' field

What am I missing?

[adarsh-ramakrishna-sfdc]

The information you shared from https://github.com/forcedotcom/ApexUnit/wiki/Integrating-with-CI-pipeline is specific to Jenkins and to the plugins available on jenkins. If you do not use jenkins, the info's "Publish HTML reports" and "Publish JUnit test result report" will not be applicable to you as different CI systems will have different ways to publish report. JUnit report(ApexUnitReport.xml ) and the HTML report (ApexUnitReport.html) are published in the jenkins using the plugins. The hard-link to ApexUnitReport.xml with in ApexUnitReport.html is just a way to provide the users with a single document using which they can have access to all the reports. A user who wants to navigate to JUnit report from the HTML report would use this link.

Does that help?

[TedHusted]

Thanks for trying, Adarsh, but we're obviously talking past each other. I'm not aware of a browser that will present an XML document as a "report", so a link from an HTML page to an XML document makes no sense to me. (Especially one that hard-codes the system path, which to me, is a security exception.) For now, I'll just post-process the file to remove the link. Again, I appreciate your responsiveness.

[adarsh-ramakrishna-sfdc]

Thanks @TedHusted . I understood your concern now.. My apologies, it took so long! .. I am interested to know what would be your suggestion to make this link useful to everyone?

[TedHusted]

[how] to make this link useful to everyone?

I'm unclear on how it is useful when used with the Jenkins plugin.

Does the plugin transform the link so that it points to a HTML report instead of the XML report?

Maybe I just need to standup Jenkins again and try it in real life.

[adarsh-ramakrishna-sfdc]

Yes, it works well with Jenkins plugin . Here's a screenshot of the web page that comes up when I click on the "Detailed Test Report" link..

[adarsh-ramakrishna-sfdc]

@TedHusted Do you still have any questions? Please feel free to re-open this thread if you have any questions. Closing for now.