As an IT architect I want to visualize reads and writes of process variables as part of its model in order to visualize the application's data flow

[larsbe]

Results of the Delphi study showed that there is great interest into organizing process data. Furthermore, validating the data flow was also mentioned as a concern. BPMN models lack the functionality of visualizing data flow (data objects are not validated and serve merely as a special comment).

Visualizing when process variables are written or read inside the process model makes the data flow of the process more accessible and can serve as a tool to facilitate manual validation of the data flow against project-specific conventions. Furthermore, it potentially helps to visualize where dependencies are used.

The idea is to lean on the ProcessVariablesModelChecker and scan the whole project (model+code) for process variable usages.

The visualization could have a filter for variable patterns (e.g. ext*, int*) and include a hierarchical selection starting on package level all the way down to individual objects.

This feature doesn't build on top of validations and consistency checks of vPAV. Therefore, it needs to be discussed whether to integrate it into vPAV or develop a separate solution.

[fkoehne]

A "separate solution" would obviously be a modeler plugin. If there is sufficient support for the idea - why not both? @cl-a-us, what do you think?

[larsbe]

I had an interesting conversation with @sdibernardo about this feature. We see it as very helpful and maybe even essential to realize the formulation of data flow rules (see. #46).

The plan is to integrate a visualization of process variables into the existing results viewer (just as a separate tab that can be selected by the user).

[StephenOTT]

@larsbe have you looked at the work being done in https://github.com/camunda/camunda-bpm-process-test-coverage/issues/32 and https://forum.camunda.org/t/script-based-unit-testing-with-coverage-reporting-spock-framework/7608. This setup tracks this kind of data, and if you want to add the points where Variables are read and written, it is a simple addition with the history service.

The grey counts for example are markers to the data flow loops: the number of times the flow moved through that task/element. Notice the sequence flows are also marked. When you run the Variable History service you can get all variable modifications for a process instance and they include the Task ID and execution Id which can be tied back to the BPMN Model element id.

[larsbe]

@StephenOTT I have looked into it and it looks pretty cool. Our approach is a bit different though. If I understand correctly, your tool tracks and visualizes data from concrete test cases. We gather our data about process variables by means of static code analysis.

I think both approaches are useful in there own ways. The pro of our approach is that it's much easier to get an overview about all potential process variable operations, however, with static code analysis we are not able to detect all potential operations. Gathering data from test cases has the benefit that you can be sure to detect all process variable operations of the given test case. However, it's not so easy to combine data from several test cases and then you still can't be sure whether your test cases encompass all potential process variable operations. So you're still left with an incomplete picture, but incomplete in a different way.

So, what about using both tools at the same time? What do you think?

[StephenOTT]

@larsbe we started looking at use cases for linter (static code analysis) as part of: https://github.com/camunda/camunda-modeler/issues/803

There is definitely a nuanced line between the static vs unit-test side of the validations and "tests" in general.

When we looked at variable usage, in practice it become something that was much easier and viable as part of a history event lookup rather than code analysis: With history event analysis you just look up the history event usage for where it occurred/which activity; But with static analysis, it become increasingly complex to detect where the changes occurred, when/if they would occur:

You end up having to detect from Result Variables, Embedded scripts, External Scripts, script loaded from third party locations from within code, Java delegates, and the variations of each of the previous. From a pure code analysis perspective: you are now dealing with setVariable, setVariables, setVariableLocal, setVariablesLocal, multiple scopes, the passing of variables between Call activities and sub processes, etc. If you were to say: well we are only going to detect "setVariable()"; then that is a simplification to the problem, but we end up with dealing with code based if conditions, loops, switch statements, various reusable functions that may be called by various scripts and conditions, etc. With all of this, I am articulating that by doing a static analysis on variable usage, you create a very limited tool that does not actually show a lot of data flow other than very simple cases.

with static code analysis we are not able to detect all potential operations.

Static review has lots of great use cases as discussed in the above github Linter issue for modeler, but applying it to execution just creates a highly limited view; We can get the full view with a automated test: Generate the test conditions automatically: As in you can provide Input vars for the start event, and vars for any User Tasks, and after thant you just need to move the token forward as part of a loop for checking: Has process ended? No, then try to complete task, no task to complete? then try to execute Job, Check if process has ended.

However, it's not so easy to combine data from several test cases and then you still can't be sure whether your test cases encompass all potential process variable operations.

This is exactly why the Coverage works and you have the reports:

Where you can track each of the paths that you follow through each step: If you are testing a BPMN and you are not sure what paths you are testing, you likely have bigger problems ;).

In the above image that is two different BPMNs with two different feature tests, can you can combine multiple pathways and flows with Data Tables / variations to track your progress across all your paths.

The tooling here is just Spock framework + spock-reports and reusable methods. You can reapply the same logic to any unit testing tooling: the point is about using the execution data rather than using static code analysis to analyze something that is execution focused/based.

So, what about using both tools at the same time? What do you think?

100%: both tooling would always be used: as discussed in the github issue for the BPMN Linter.

edit: Another issue/variation to this reporting issue which i am dealing with at the moment is the nuances of "save points". Where someone might save a variable multiple times during a single transaction: but its really only the final save that matters. So this is a example where execution vs static code analysis will show different results: static code would show multiple save points, but really its one save, as the previous are overwritten within the transaction/save point.

[StephenOTT]

@larsbe here is a example of coverage showing which activities have variable writes:

Under the hood this is what is going on:

Collection<HistoricDetail> variableHistory = processEngine.getHistoryService().createHistoricDetailQuery()
                                                                                .processInstanceId(processInstanceId)
                                                                                .disableBinaryFetching()
                                                                                .variableUpdates()
                                                                                .list()

        List<Map<String, Object>> activityVariableMappings = variableHistory.collect { historyItem ->
            [('activityId'): processEngine.getHistoryService().createHistoricActivityInstanceQuery()
                                            .processInstanceId(processInstanceId)
                                            .activityInstanceId(historyItem.getActivityInstanceId())
                                            .singleResult().getActivityId(),
             ('variableInstance') : historyItem.toString()
            ]
        }

And then we are just looping through the data to add the proper overlays. This also has the extra benefit of being able to add all of the logic about the actual variable change.

So here is the data of a activityInstanceVariableMapping that was collected for the third BPMN image where two variable edits are being showing

[
 [activityId: Task_1237h9u, 
    variableInstance: HistoricDetailVariableInstanceUpdateEntity[
            variableName = dog, 
            variableInstanceId = 11, 
            revision = 0, 
            serializerName = null, 
            longValue = null, 
            doubleValue = null, 
            textValue = cat, 
            textValue2 = null, 
            byteArrayId = null, 
            activityInstanceId = Task_1237h9u: 10, 
            eventType = null, 
            executionId = 7, 
            id = 12, 
            processDefinitionId = 7, 
            processInstanceId = 7, 
            taskId = null, 
            timestamp = Mon Jun 25 16: 08: 57 EDT 2018
        ]
    ],
 [activityId: Task_0hed6o6, 
 variableInstance: HistoricDetailVariableInstanceUpdateEntity[
     variableName = dog, 
     variableInstanceId = 11, 
     revision = 1, 
     serializerName = null, 
     longValue = null, 
     doubleValue = null, 
     textValue = 123456789, 
     textValue2 = null, 
     byteArrayId = null, 
     activityInstanceId = Task_0hed6o6: 26, 
     eventType = null, 
     executionId = 7, 
     id = 27, 
     processDefinitionId = 7, 
     processInstanceId = 7, 
     taskId = null, 
     timestamp = Mon Jun 25 16: 08: 58 EDT 2018
    ]
]
]

So as a result we can do all sort of nice overlays to show the variable data that was being changed (in this case it was the same variable being overwritten in the second script task.

Now its just adding the extra interaction js and css: https://github.com/bpmn-io/bpmn-js-examples/tree/master/interaction, all of the data is already being provided and is always accurate.

another zoomed version:

[larsbe]

@StephenOTT

You end up having to detect from Result Variables, Embedded scripts, External Scripts, script loaded from third party locations from within code, Java delegates, and the variations of each of the previous. From a pure code analysis perspective: you are now dealing with setVariable, setVariables, setVariableLocal, setVariablesLocal, multiple scopes, the passing of variables between Call activities and sub processes, etc. If you were to say: well we are only going to detect "setVariable()"; then that is a simplification to the problem, but we end up with dealing with code based if conditions, loops, switch statements, various reusable functions that may be called by various scripts and conditions, etc. With all of this, I am articulating that by doing a static analysis on variable usage, you create a very limited tool that does not actually show a lot of data flow other than very simple cases.

You are very right that static code analysis always has some limitations, but I'm not so sure if this leads to a "very limited tool". Many of the different ways to manipulate process variables you mentioned (and others) are already checked by vPAV, it's just a matter of showing this information to the user. If you look at the changes made in https://github.com/viadee/vPAV/pull/54 you'll see that it's just about exporting the data and integrating it into the results viewer app. So it was an easy change for us that definitely has value to the user.

We also think that most of variable operations are easy to detect. Most of the cases we can't detect are rare cases in practice. So what we get is kind of an 80/20 solution (which is a very pessimistic estimate I think) that is more than enough for the use case of getting an overview. Just to clarify, we don't claim to give you the complete truth, but we can give you hints on some issues that exist in your process application.

We can get the full view with a automated test: Generate the test conditions automatically: As in you can provide Input vars for the start event, and vars for any User Tasks, and after thant you just need to move the token forward as part of a loop for checking: Has process ended? No, then try to complete task, no task to complete? then try to execute Job, Check if process has ended.

Here, I have to disagree. With unit tests your view is limited to what happens in unit tests. So, if you don't have any unit tests, you don't have any data about process variable operations. Generating test conditions is a good measure to mitigate this problem, but for a full view you'd need to test every possible combination of inputs, which is practically impossible. So you also end up with a limited view...

I'm not so sure if arguing which approach is superior to the other is the best way to discuss this issue. I think there are use cases for both approaches. Just to give you an example: think of an existing process application that historically grew with low quality standards (few/no tests). Then having a tool like vPAV gives you great insight into what is going on. It could also be the starting point for creating test cases like yours that give you more details and the actual truth about what's happening in these test cases (and if the test cases represent real cases then you're very near the actual truth).

[fkoehne]

One more scenario that we can think through: A process under active development, that is not yet executable. From my experience, it is often difficult to get adequate mocks for the environment, that is being orchestrated early on. Here, static analysis the only viable option available, but will probably lose some of its value, the more advanced your test setting becomes.

[StephenOTT]

Hey!

One more scenario that we can think through: A process under active development, that is not yet executable. From my experience, it is often difficult to get adequate mocks for the environment, that is being orchestrated early on. Here, static analysis the only viable option available, but will probably lose some of its value, the more advanced your test setting becomes.

@fkoehne is the process is not executable in the sense that it is finalized, It surely can be easily flipped to "manual tasks" wherever there are incomplete executions but the visualize task is still present. Execution from are use of the scripting has been simplified to the point where you really just need a start event. The process does not actually have to be fully executable/completable.

Here, I have to disagree. With unit tests your view is limited to what happens in unit tests. So, if you don't have any unit tests, you don't have any data about process variable operations. Generating test conditions is a good measure to mitigate this problem, but for a full view you'd need to test every possible combination of inputs, which is practically impossible. So you also end up with a limited view.

i think this is just a difference of "view" on what you are trying to visualize: 1. visualize all possible routes at once (static view), and leave the interpretation of those routes up to the human. 2. Define the possible routes and evaluate based on those defined routes: if the human can visualize all possible routes and make sense of the data, then they can surely define the possible routes at the highest of levels (not saying you need to define micro pathways, just broad strokes at the gateways and where additional data is submitted into the system; after which everything else can move forward automatically through job and user task detection.

---

I am not saying the static analysis is bad (the BPMN Linter discussed in modeler repo is exactly this purpose). The specific to what we were talking about are specific scenarios such as "data flow" / save points where static analysis becomes more useless other than knowing where there are specific keywords in the code. If we have a Script or delegate that executed a chunk of code that did a variable save; and that code that called based on a JSON file configuration that was loaded into a process variable, and that specific file was loaded at runtime through a inputstream from a Deployment Resource: all that abstraction is missing from the static analysis because the files are not available for analysis. And when we add multiple Data point saves that are conditionals (If this then save, else dont save), those conditions are not evaluated during static analysis because they are execution based. So i am just articulating that there are specific items such as the data flow that having the execution is what makes the information "actionable information", otherwise it is more of a "Show me where there are possible saves throughout the bpmn", rather than "show me where/when i am saving data" for each of my pathways. 👍

[larsbe]

The specific to what we were talking about are specific scenarios such as "data flow" / save points where static analysis becomes more useless other than knowing where there are specific keywords in the code. If we have a Script or delegate that executed a chunk of code that did a variable save; and that code that called based on a JSON file configuration that was loaded into a process variable, and that specific file was loaded at runtime through a inputstream from a Deployment Resource: all that abstraction is missing from the static analysis because the files are not available for analysis.

Your second example sounds like a design for a process application that's very hard to understand and to maintain. In this case the data model of your application depends on runtime data and I'm not sure you'd actually want that. Besides, such applications are much harder to test especially in isolation: if your data model depends on runtime data, then the data flows from your tests stop to represent reality as soon as your test data differs from your runtime data.

From your first example I take that you're not fully aware of what our analysis actually does and is capable of. I encourage you to try our DEV version and play around for a bit to see what we actually capture and what we don't. If you have ideas on how to improve it, we're also very happy to hear about them. If you still think about it as useless afterwards, I'm ready to acknowledge the difference in opinion and leave it at that.

[fkoehne]

@StephenOTT - We would really like to have your input regarding the performance of the dev branch vPAV on your projects to see, if we catch >80% of the variable operations (across model, scripts and Java code) as Lars indicated ;).