Simple webinterface for build results

[mvriel]

Design Suggestion

Showing build results can be done in two ways:

1. A Dashboard that gives a high-level overview of the each build step's results.
2. One or more pages that can be opened separately where you can inspect the detailed results of each build step (or series of build steps).

My proposal is to re-use the idea of Parsers to have 'Report' Parsers which can generate artifacts (usually HTML files) and instruct FutureCI to link to those generated artifacts from the Build Result page.

This linking can be in one of two forms:

1. a button to download or externally navigate to the aforementioned artifact (think build logs for example)
2. a tab in the build detail page that will show the generated artifact by including it using AJAX or using an iframe (which is more convenient for multi-page artifacts such as Code Coverage)

Development for this feature can be divided into two stages:

1. Showing detailed build results using tabs / buttons
2. Showing a Dashboard

Because a dashboard sounds nice but is a large investment of work (since it is (usually) convenient to provide a layout for your dashboard widgets and include a widget system) I suggest to start with the functionality to render detailed result pages (artifacts) and embed them in the build result page using tabs or buttons.

In this phase we do not yet implement support for buttons; this should be added once it we have downloadable artifacts that we want to expose.

Phase 1: Build results using tabs

A Build Result in the broadest sense is a file that is created from the output of a Task which is stored in the /var/builds/ folder. An example of this is the report.xml file that is generated by Codeception.

What we need to do is provide a way to expose these artifacts in a meaningful way to the end user. This may mean that a human-friendly representation needs to be rendered; this can be done using a specialized 'Formatter' similar to the 'Parsers' that already exist but with their own section in the build file.

A Task should support multiple Formatters and these are to be executed in sequence so that we can allow for them to generate Build Artifacts that can act as Reports. As an example of this we can have a CodeceptionHtmlReportFormatter that takes the generated report.xml, pass it through twig with a special template and get a report.html file containing a summary of all test results.

Codeception has an HTML report for tests but because we want to be able to match the design of FutureCI we will need to render our own report (not to mention that the Codeception HTML report is ugly as f*ck).

With this type of Report we can have a new tab in the build details page where you can view the results. For performance reasons and because Twitter Bootstrap supports this out of the box I recommend loading these results using AJAX. This does have one downside: the artifacts are currently not world accessible. In order to solve this a new Action should be added to a controller with which a build artifact can be exposed.

An advantage of using an Action is that at a later stage we can introduce authorization where only signed in users with the right permissions can view certain build artifacts. If we expose the build artifacts directly without proxying them through an Action this will be much harder to accomplish.

With our design we should keep in mind that at a later stage we will need to have different strategies for tabs because you could also choose to have a tab that doesn't use AJAX (because there is no physical artifact to load) or uses IFRAMES because the generated artifacts have their own CSS and JS files (such as Code coverage reports).

• [ ] Add an Action Controller that can expose the artifacts in the /var/builds/*/* folder as these elements life outside the web root. (no authorization is added)
• [ ] Add a tabs element in the metadata array of the Result object
• [ ] Add AJAX tabs to the app/views/build/show.html view which uses a link in the tabs element of the Result's metadata to dynamically load an artifact directly into the tab.
• [x] Add support for formatters with tasks in the build configuration
• [ ] Add HTML Report Formatter for Codeception test results:
  • [ ] Create report.html with listing of tests and whether they have failed or errored
  • [ ] Add an AJAX tab to the tabs element of the metadata array in the Result object pointing to the report.html artifact.

[mvriel]

Phase 2: Add support for Code Coverage

More details should be added; though it is rather similar to the above.

• [ ] Enable Code coverage in Vagrant machine (xdebug.code_coverage is set to 0 and should be 1 in xdebug.ini)
• [ ] Add IFRAME tabs to the app/views/build/show.html view which uses a link in the tabs element of the Result's metadata to dynamically load an artifact into an iframe of the tab.
• [ ] Add Code Coverage flag to CodeceptionTask.
• [ ] Add Report Parser for Code Coverage results.

[mvriel]

@jaspernbrouwer @ramondelafuente can you review the suggested approach in my first comment? The addition for code coverage should be added in a later PR but I think you also get the gist of that :)

[jaspernbrouwer]

This all sounds good to me!

The ResultParsers are meant to read a result/report file(s) generated by a Task and turn them into a Result object that can be understood by the rest of the application. It basically boils down to determine if the task as succeeded, failed or borked.

Other information (like individual test results of Codeception) should indeed also end up in the Result object (or maybe a different Result object), which can be used to create some sort of readable view.

I suggest to name these last things Formatters (or something similar), not ResultParses. I'd like to keep the distinction between ResultParses (which turn raw results into understandable results) and Formatters (turning understandable results into something readable).

One last thing: I don't think code-coverage in the Poolz project has any meaning. We'll probably never reach anything near to a 100% ;)

[mvriel]

Updated approach to include formatters.

Regarding code coverage: to me the goal for code coverage is to get an indication of what is tested and to be able to spot crucial areas that might need coverage. Also: while developing tests I check with the code coverage to see if I have not forgotten an important scenario.