jejacks0n / teaspoon

Teaspoon: Javascript test runner for Rails. Use Selenium, BrowserStack, or PhantomJS.
1.43k stars 243 forks source link

Export HTML report with --export #165

Closed nilbus closed 10 years ago

nilbus commented 10 years ago

This PR implements #121.

The Teaspoon reports can be exported as a static HTML page. This requires that wget is installed.

teaspoon --export="output_directory"
teaspoon --export # output directory defaults to './teaspoon-export'.

To accomplish this, wget downloads from the teaspoon server the HTML and any assets the page needs to load. The page (by default) is downloaded to ./teaspoon-export/suite_name/index.html.

Specs have been added to cover the new functionality. This is also manually tested and working in my JRuby app, which mounts teaspoon at a nonstandard route (/web/teaspoon).

Cookie session storage is not available when viewing static html files via file://. Therefore instead, the HTML reporter uses localStorage when available. If localStorage is not available, it falls back to using cookies.

Please give me any feedback you have. If you're happy with this implementation, I'd love to see this released as a new gem version soon. Thanks!

jejacks0n commented 10 years ago

Thanks. =) Can you give me some background so I understand why you want this? Before I dig into it, I want to understand what this provides or solves for you.

nilbus commented 10 years ago

Good question. In addition to running the tests, our Jenkins CI builds create build artifacts that give us an HTML representation of each of our test suites' results (rspec, java dbunit, etc., and now teaspoon) and save them for later viewing. This is useful because our test suites run concurrently, and some generate lots and lots of output in the build log that is not usually helpful when trying to examine the test results. Teaspoon was the odd one out that was only able to output the test results to the CI build log and not an HTML file.

jejacks0n commented 10 years ago

Can I ask why none of the existing build artifact/formats worked? eg. TeamCity, jUnit, or TAP?

nilbus commented 10 years ago

I really like the HTML format that Teaspoon already provides. I considered trying to recreate something similar, translating from TAP data, but I have a hard time trying to justify the work to recreate that when I can instead capture what is already provided.

There's an important difference too in that the HTML export is an active test, whereas HTML output that translated from TAP data would be static. Since the export is actually running the test against the javascript code on each load, it can be investigated in different browsers right on the export page rather than having to check out the right branch and run teaspoon locally.

Mostly I just wanted artifacts that look and act like the Teaspoon web interface itself.

jejacks0n commented 10 years ago

I played around with this concept in the 0.8 branch, and I think we can do it more efficiently using phantomjs/selenium instead of wget.

I'll see if this can go in cleanly.

I had a question though -- you mention saving all the assets -- is this needed, or can they be re-served from the CI box? As in, I would rather just save the index.html and not all the assets that were loaded. I'll have to look at what you did there.

nilbus commented 10 years ago

They cannot be re-served from the CI box. Every build has a different branch or commit with potentially different versions of assets. Once the CI build is complete, not only does the server stop, but the build tree is deleted. This is why we need to save a copy of all the assets.

As far as I can tell, phantom has no mechanism for downloading a page's assets and rewriting the URLs on the page so that it can load the assets locally. You would have to implement that yourself, which would be quite an undertaking.

jejacks0n commented 10 years ago

Getting this into 0.8.. thanks. =)