Reports must generate JSON instead of HTML

[robhrt7]

For the sake of flexibility, it's essential to provide raw report data, that could be then styled any way user wants.

[ishubin]

Thanks! I remember we discussed it yesterday. One thing I remembered it should be implemented only after #145 "Redesign Layout report" is finished. Also we will have to think through the JSON structure that will be useful for everyone. Perhaps will start just with dumping to json the LayotReport object and there we shall see what is needed more.

[erquhart]

+1

[hypery2k]

should not be that big problem. I've already started redesigning the report functionality, see #188

[ishubin]

Implemented in release-1.6 branch. At the moment it creates a "report.json" test overview report. And for each test separately it creates json detailed report. Also from now on html reporter is based on json reports. That makes it easier to maintain and improve

[hypery2k]

There are some issues with rainbow4j see https://github.com/galenframework/rainbow4j/issues/2 and some of the browser tests are failing: https://martinreinhardt-online.de/jenkins/job/GalenFramework_integration/browser=chrome/2/console

[hypery2k]

some feedback from production: The report generation is working and looks great ;) I'd like to add tag support. Are you currently done with the redesign?

Some minor remarks.

• You renamed the method LayoutSpec.getErrorMessages() to LayoutSpec.getErrors(), any reason for that?
• If an element could not be found it would be nice to mark it in the report
• Would be helpful to add a message to the JSON which shows up in the report for errors and passed checks

[ishubin]

@hypery2k I don't really think there was any reason. Its just that I first came up with JSON I wanted to generate and then based on that renamed a lot of fields. Are there any problems on your project caused by this rename?

I am almost done with redesign. There still minor thing left but that is on the test report level. The report overview is finished. Basically you can work on that. We shouldn't have too much conflicts in this area. You can just implement tags as a list of strings (like in TestNG). This means that the "tags" will be presented in JSON test overview report. And based on that the html can also be rendered in two tabs: tests table as it is now and tags view. Do you have already some ideas on this?

[ishubin]

@hypery2k

If an element could not be found it would be nice to mark it in the report

As I understand you are describing the situation when element was not found in the DOM, right? But what should we show in this case other than just a screenshot and an error?

Would be helpful to add a message to the JSON which shows up in the report for errors and passed checks

You mean like a list of errors in the layout report? I think it is a good idea. That will help to aggregate all errors in one place. Will implement that.

[hypery2k]

No i mean when the locator is not found, see sample here:

[hypery2k]

I will start with adding the tag support next week, this week I'm too busy for this. Maybe on the weekend

Regarding the methods: In version 1.5.x there were some error details, but with 1.6. branch they seems empty. I will investigate on that

[ishubin]

@hypery2k What you have shown on the screenshot is a bug. I will fix it. Thanks!

[hypery2k]

thanks for that. I can retest and will add some tests later for that

[ishubin]

@hypery2k I have pushed the fix for absent objects.

[hypery2k]

I've implemented basic tagging support for the reports: The best of this is, it reuses the existing tags. So should be shown up automatically ;)

Next step should be to enabled grouping of tests, see above. So collapsing and expanding.

Additionally filtering of tags would be nice

[hypery2k]

one remark: Besides the rendering of the JSON report data within the HTML we should add also a "pure" JSON output report file!

[ishubin]

@hypery2k What do you mean by "pure" JSON output? At the moment there is already an output of json report in release-1.6 branch https://github.com/galenframework/galen/blob/release-1.6/src/main/java/net/mindengine/galen/GalenMain.java#L356

you can just provide --jsonreport path/to/json/report arguments in command line

do you want something else there?

[hypery2k]

I mean just store a plain .json file in the report directory

[ishubin]

Ah, ok. I think it is a good idea