Merging multiple snapshots into a single HTML report

[LucaMozzo]

In Java Spring JUnit tests I collect the snapshots at the end of every test with

coverageBuilder.coverageSnapshot(processInstance);

and then I have a final action to save all the HTML reports

  @After
  public void saveCoverageSnapshots() {
    coverageBuilder.saveCoverageSnapshots();
  }

However I was wondering if there is already a functionality in place to merge all those reports into a single HTML file (so that a single coverage report is generated for every class, rather than for every process instance).

If it is already there and you could explain that to me, I'm happy to add an example to the readme too

[StephenOTT]

No feature like that is currently implemented. My thinking was: in cases where you needed a single file or report (like the Spock report image in the readme), you would create a HTML file with multiple embeds/iframes that are losing the needed HTML files.

Can you give me some more usage reasons for needing a single file?

Side note: we could add the ability to get the coverages as a list of input streams. And you could save these into a single HTML file.

[StephenOTT]

Another item about my original thinking: there are multiple files because the snapshots were designed to be used throughout the process, not just at the end. So there are multiple files to show the different shots of the single process.

If you have a single file, are you looking to add additional text like headers for each process/snapshot?

[LucaMozzo]

I believe that what you've done already is really good to check whether the path was correct or not and if it terminated correctly. My usage scenario is to generate a report that shows all the tasks that have been tested, and therefore determine if more tests are needed to achieve 100% coverage. Of course that single report won't take into account all the possible paths but just the task completion (a bit like the heat maps in Camunda Cockpit). I believe that summing the number of executions for each task (in the green box you have already) should be sufficient for this feature.

What do you think?

[LucaMozzo]

However the idea of stacking them one on top of the other in the same file doesn't sound bad either; the only problem is that in my current project we have about 70 tests and they can are likely to increase even more, and reading from 100 html files or stacking them on top of each other can become tedious.

Another question on this matter: is there a reason because there are 2 copies of the same Camunda model in the report at the moment?

[LucaMozzo]

I noticed that it's possible to generate coverage data given a ProcessInstance. Technically it would be possible to accumulate the number of times each task has been completed (CoverageData.activityInstancesFinished field) and generate the report based on that data

[StephenOTT]

@LucaMozzo

I believe that summing the number of executions for each task (in the green box you have already) should be sufficient for this feature.

So to ensure i understand: you are looking to get counts of the number of executions of each task/activity?

The grey box that appears to the bottom right of the activities is the count of executions.

So that would give you the sum on a single process excecution, But would not give you the total across multiple executions.

---

Multiple execution views is a interesting use case....

Maybe Something like the ability to provide multiple process instance objects and get a Summation view? Could even add the heat map visual as a option.

---

I noticed that it's possible to generate coverage data given a ProcessInstance. Technically it would be possible to accumulate the number of times each task has been completed (CoverageData.activityInstancesFinished field) and generate the report based on that data

Correct. Thinking the same sort of thing. I am thinking it would be a new set of functions that aggregate the data and create a single summation as described above.

[StephenOTT]

@LucaMozzo can you provide a use case scenario: Like a mock BPMN image showing the flow and describe how you are executing and the type of result you want to create/validate against? (What a good result looks like and what a bad/useless result looks like)

[LucaMozzo]

Ok let's pick a basic example. I have a diagram that looks like this

I have a class with tests for each possible outcome, since I want to ensure that all cases work fine

class Tests {

@Test 
void Task1Task2&Complete() {.  //test for red path
...
coverageBuilder.coverageSnapshot(processInstance);
}

@Test 
void Task1Task3&CompleteWithErrors() {  //test for green path
...
coverageBuilder.coverageSnapshot(processInstance);
 }

  @After
  public void saveCoverageSnapshots() {
    coverageBuilder.saveCoverageSnapshots();
  }
}

So basically every time I call coverageSnapshot it "accumulates" the number of task executions and when I call saveCoverageSnapshots() it generates a single html report with the accumulated task executions. For instance in the first test I'll have Task 1 : 1 execution Task 2: 1 execution Task 3: 0 executions

and after the second test

Task 1 : 2 executions Task 2: 1 execution Task 3: 1 execution

And a single diagram is exported with the values specified above.

If instead I wanted to generate a single report for every test (like it is working now) I would save the snapshots after every test:

class Tests {

@Test 
void Task1Task2&Complete() {
...
coverageBuilder.coverageSnapshot(processInstance);
coverageBuilder.saveCoverageSnapshots();
}

@Test 
void Task1Task3&CompleteWithErrors() {
...
coverageBuilder.coverageSnapshot(processInstance);
coverageBuilder.saveCoverageSnapshots();
 }
}

This is an example, and the way I thought the library was working by looking at the available methods. Of course in my example it doesn't make sense since it's really simple, but when you have 10+ workflows with about 5 to 10 tests each, you would be able to generate a single report for every workflow (like in the first snippet of code) ending up with 10 reports instead of 100.

Does this example help you understanding the scenario (on a bigger scale)?

Having a single file for each test is more informative because you can see each of the paths that have been covered, but on a large scale it's not possible to look at every single path.

The idea of the heat maps sounds good, but I don't know how much work it requires, but having the general functionality in place with just the numbers like in your photo above would be enough

[StephenOTT]

@LucaMozzo in your example (even at scale), why would you want the sums? The sums seem to be a "cross-check", but they could easily throw a false positive due to other weird errors occurring.

Would you not want a Visual Per method? So you have a report for each method you are executing and thus validating visually each path?

class MyBPMN123Tests {

@Test 
void Task1Task2&Complete() {
   ...
   coverageBuilder.coverageSnapshot(processInstance, "task1-task2-complete");
}

@Test 
void Task1Task3&CompleteWithErrors() {
   ...
   coverageBuilder.coverageSnapshot(processInstance, "task1-task3-complete-with-error");

 }

@AfterClass
public static void teardown() {
   coverageBuilder.saveCoverageSnapshots();
}  
}

With the above you would generate files with names of each method and only generate the snapshot. I am assuming each @Test is a compartmentalized scenario and you are doing cleanup inbetween tests? aka you have new process instances per @test?

You would end up with a folder called something like org.mycompany.MyBPMN123Tests and files called 0_task1-task2-complete.html and 1_task1-task3-complete-with-error.html

[LucaMozzo]

It's not much for "better visualisation" but more for admin. For instance you can't create a QA report with 70 screenshots attached to it, but you rather have a less informative diagram, but you can see at a first glance if all tasks have been covered in each of the workflows. Of course for a more in-depth qualitative testing the current method would be the best.

[StephenOTT]

@LucaMozzo okay so for the "QA Report" its more of a quick check without the need for further analysis, but you are accepting the risk that it could be a false positive?

[LucaMozzo]

Yes I am aware that the fact that, even if all tasks are completed, some paths might not have been covered. That just needs to be an approximate view

[StephenOTT]

@LucaMozzo I have not had time to write the implementation for this, but I do have a solution for you which you can write with some post-class code:

1. I released v0.14. In that version there is a setter and getter for the coverageSnapshots property. You can now access this in your javaclass due to the addition of the getter. coverageBuilder.getCoverageSnapshots()

2. It will return a hashmap of <String, CoverageData>. Using this you can post-process all of the coverage data snapshots into a single Coverage Data object. Essentially you are doing a Merge/addition (would be a good use case for using a java 8 Stream.) of each of the coverageData objects' data/properties and creating a new coverage data object which is new CoverageData(); (io.digitalstate.camunda.coverage.bpmn.CoverageData;) You can populate that object with the new aggregated values.

3. then do something like coverageBuilder.setCoverageSnapshots(["aggregated-data": theNewCoverageDataObject])

4. then coverageBuilder.saveCoverageSnapshots()

This should result in a single file being saved with your aggregated data, with a file name similar to 0_aggregated-data.html.

[LucaMozzo]

@StephenOTT Thanks a lot, that's great!!