Define strategy for implementing the E2E test suite for the new DMN Editor

[jomarko]

Edited: Multiple rounds of feedback incorporated.

Initial requirements:

• Easily create new tests.
• Easily understand what a test case is testing.
• Easily understand why a test broke.
• Easily run single or multiple tests locally.
• Easily debug tests that are running locally.
• Easily update tests that broke during development.
• Easily associate an issue to a regression test.
• Ability to test multiple browsers.
• Rely on tests not being flaky.
• Easily monitor the health of our test suites, looking at reports and metrics.
• Internal refactoring that don't have impact on anything visible shouldn't require tests to be changed.
• Rely on tests being as fast as possible, without sacrificing anything from the above.
• To some level, assert compatibility of related tooling (SceSim editor, DMN Runner, Dev Deployments, Sandbox and VsCode included models) with the the DMN Editor

Things we'll do that will get us there:

• We will follow the example from the new test suite of the Boxed Expression Editor.
  • So for example, we should follow same folder structure, naming of files and scripts
• We will have mechanisms to associate a test case with an issue.
  • The issue key should be the only needed piece of information to check, if that issues is somehow related to the test suite (i.e. there is a regression test, blocked test ...)
• We will have 3 retries for each test, that will only fail if failing more than 3 times, minimizing flakyness.
• We will have PlayWright run tests on Firefox, Safari and Chrome, on Windows, Linux, and macOS.
• We will have single-line commands that anyone will be able to run on their machines for running either
  • 1. all tests,
  • 2. a single test,
  • 3. a set of tests.
• We will have a way to debug tests during execution, placing breakpoints directly in our IDEs of choice (VS Code, Webstorm etc).
• We will have a way to look at the test playback from a CI build, or from a local run.
• We will have the tests output their results in JUnit XML format, uploading them to Buildkite on community.
• In general, we will not create one DMN file for each test case. Instead, each test will create its DMN from scratch, making use of highly optimized functions that will execute common actions, like:
  • const decision = addNodeFromPaletteToPositionXY({ type: "Input", x: 100, y: 100 });
  • const input = addNodeFromPaletteToPositionXY({ type: "Decision", x: 100, y: 50 });
  • connectNodes({ from: input, to: decision });
  • selectArea(start: { x: 0, y: 0 }, end: { x: 300, y: 300 });
  • assertUsingScreenShot() // or similar
• In general, we will not create giant tests. Instead, we'll create the most focused tests possible, testing one thing at a time.
• In general, big/complex models won't be created from scratch. Instead, if they're really needed they'll have a distinguishable name and very nice documentation around it. E.g., "Traffic violation example" or something like that.
• We will not use existing models from external sources. Instead, we'll define our own models either in our tests themselves or use DMN files inside kie-tools.
  • models stored in kie-tools should follow test structuring described bellow
  • also models intended for checking DMN Editor compatibility with the rest of the tooling should be easily distinguishable
  • for the points above, we should use probably some folders or prefix/sufix naming

Structure

Test types:

• DMN features
  • E.g., A Decision node can connect to another Decision node via information requirement.
• Our features
  • E.g., Decision Service nodes are always visually behind other nodes.
  • Creating a new DRD auto-focuses on the DRD name automatically for easily renaming it.
• Regressions
  • E.g., kie-issue#XXX: Lorem ipsum ...., possiblity with a DMN file associated with it.
• Non-functional
  • E.g., unnecessary renders, performance, etc.

NOTE: This is not black & white, though, some tests might be in a gray area between two or more types. And more types might appear as we progress.

Asserting strategy:

• In general, UI tests will focus on screenshots and elements being present on the screen.
• Regression tests can rely on other assertion techniques, like looking at the underlying DMN model or even the XML itself.
• Negative assertions are usually best avoided. E.g., "assert that something did not happen" or "assert that something is not present on screen"

Remarks:

• Duplication is not always bad. Each test gets to be isolated from one another.
• Make use of well rounded primitives for reusability and readability.
• Tests being 100% detached from the source code allow for greater freedom when developing.
• Having a great set of primitives will allow other components to use them when interacting with the DMN Editor inside their own tests. E.g., KIE Sandbox testing Dev deployments, or multi-tab editing, for example.
• Somethings are better tested as unit tests. We should be very considerate of choosing the testing strategy for something. For example, the validity of a FEEL name should be tested by unit tests, but the fact that the UI renders invalid names in red, should be an E2E test.

Things we won't do:

• Mock too much to simulate tests that should actually be somewhere else: E.g., multi-file editing on KIE Sandbox / VS Code Extension
• Test running the DMNs (that will be a separate test suite..., not associated with any component in particular.. like @kie-tools/dmn-e2e-tests, or even in another repo!)

---

Action plan:

1. [IN PROGRESS] Exploratory phase: phew test cases, predefined, will be used to create some ground work.
   • Creating the StoryBook and migrating the Dev Webapp there.
   • Defining basic tests for dragging nodes into the diagram and making connections between them.
2. Expansion phase: create more tests to identify problems at a larger scale.
   • Refine our primitives, refactor the first tests, get a better sense of how things are going to look like once the test suite is bigger
   • Start integrating with the CI, improving documentation etc.
3. Maturity phase:
   • Finish adding the initial tests we want, improve pain points and smooth edges out.
   • Ask for feedback, share the progress.
   • Document how other people on the team can use and interact with the test suites.

Next steps:

- Monitor the quality of our tests, see what is working well and what is not.
- Keep an eye on PRs to make sure everyone is adding tests when changing things.
- Help propagate the testing culture to the team, offering help and acting as owners of the test suites.
- Continuous work. It doesn't stop here.

[tiagobento]

@jomarko Hi! Looking good so far. A few comments from my end:

1.

I like how you defined "Goals" and "Non-goals" in the description here, it's important to know where we draw to line and stop bending and twisting to test something that should really be somewhere else. One thing I'll add to the sections you created, is that I think we would benefit a lot from separating functional from non-functional requirements. Let me explain it better.

Testing whether or not a node can be correctly dragged from the Palette to the Diagram is very close to testing whether or not the copy/paste mechanism works, but testing that nodes components are not unnecessarily re-rendered when editing the Diagram, is not. We have a lot of "unsaid" non-functional requirements for the new DMN Editor, and I think it would be great if we could map them as part of the test suite too. WDYT?

2.

Integration with External models is really complicated and needs to be thoroughly tested, and although I understand where you're coming from with saying that these tests belong on KIE Sandbox / VS Code extensions, and even agree that there should be some sort of testing for this features on their test suites, I think the DMN Editor test suite should have a lot more tests covering External models. Right now, the entire API of the DMN Editor is defined by the Props of the DmnEditor component, which include some properties related to External model integration. I.e., onRequestExternalModelByPath, onRequestExternalModelsAvailableToInclude, externalModelsByNamespace. The last one, externalModelsByNamespace, really is where the DmnEditor component gets everything it needs from External models, and there are so so many things to test, that I think we should "mock" multi-file workspaces and test them as part of the DMN Editor suite.

Note that for KIE Sandbox and VS Code Extensions, we'll also have dmn-editor-envelope at play, and it has some logic related to multi-tab editing etc, that's what's really not in scope for the DMN Editor suite, IMHO.

3.

Although I agree that we'll need some DMN files to be versioned and opened in the DMN Editor, I think we should try and aim for creating the majority of our fixtures during the tests themselves. WDYT?

4.

The "Creating custom DMN types from the java files" feature is not present on the new DMN Editor yet. See https://github.com/apache/incubator-kie-issues/issues/782

[jomarko]

Thank you @tiagobento let me work on second version of the strategy I think we should have for testing and I will try to incorporate your comments.

[jomarko]

What

We should introduce new e2e test suite for the dmn-editor module. It means we should focus on its key features, while we should not repeat the logic tested as part of other test suites.

Strategy

The most important question probably is the amount of test and the level of details we should focus on.

We should probably avoid short test scenarios like:

• open diagram, move node, close diagram
• open diagram, rename node, save
• open diagram, move node, undo, close diagram

The reasoning behind is this:

• similar actions are usually also parts of more complex scenarios. If we imagine modeling large diagram from a real domain, we can be sure all actions from the above will be included.
• the shorter the scenario is, the higher chance is the test should be rather unit test or mocked test instead of the end-to-end test.

What are then the scenarios for the end-to-end testing? We can split the scenarios into three categories.

01 Model from scratch

We live in the age of the AI, however still, some DMN models, need to be created from scratch by real human users. But from the dmn-editor test suite point of view, what such set of models is. Or let’s reword the question. What is the set of models, the dmn-editor test suite will guarantee, they can be created. Is it a set of models present:

• in kogito-examples?
• In community documentation?
• In new, for now non existing storybook documentation?
• In ticket where new templates are discussed?
• …. ?

Regardless, what the set of models we agree is needed to test, we should not try to test from scratch ‘every model from the universe’, that we want to support, because it would mean a duplication of testing the same:

• adding nodes
• connecting nodes
• creating new data type
• ….

We should minimize the set of tested models but trying to be sure:

• every node type was used in test suite.
• simple custom data type was used in test suite.
• data type with constraint was used in test suite.
• structured data type was used in test suite.
• …
• And more from the ‘Key Features’ list

We should also minimize the overlap with other test suites. For example, for modeling nodes content, i.e. expression, we should not create expression from scratch in the dmn-editor test suite. We should simplify this part using some feature like ctrl-c + ctrl-v probably. Reasoning behind this is creating expression from scratch should be part of the boxed-expression-component test suite.

02 Reusing model

Second category is a complement to the first category. Or let’s say it depends on the first category. Here users reuse logic of other models. On high level, we can say it is two phases procedure.

• import a model with logic to reuse.
• reuse imported model’s logic.

However, as for the first category, what are examples of such models, that we want the dmn-editor test suite will guarantee, they can be always created? Do we find them:

• in kogito-examples?
• In community drools documentation?
• In new, for now non existing storybook documentation?
• In ticket where new templates are discussed?
• … ?

Regardless of the answer for the above, the situation is more complicated comparing 01, because one test now needs to operate with more models. Right now, the entire API of the DMN Editor is defined by the Props of the DmnEditor component, which include some properties related to External model integration. I.e., onRequestExternalModelByPath, onRequestExternalModelsAvailableToInclude, externalModelsByNamespace. The last one, externalModelsByNamespace, really is where the DmnEditor component gets everything it needs from External models. This should allow us to do testing of including models, as part of dmn-editor test suite, however we will be not able to avoid mocking.

That is why we need to introduce tests for including models, also into other test suites:

• vscode-extension tesuite
• online-editor test suite

03 Non-Functional tests

01 and 02 are both functional tests. However wee need to test also non-functional aspects as part of the dmn-editor test suite.

• We should check components are non-necessarily re-rendered. Debug mechanism was introduced as part of PR that should help us to do it.
• We should check opening large models doesn't take too long
• We should check opening other vendors models is not an issue ???

Supported models

We described some models in 01 From Scratch and 02 Reusing model sections. Let’s start to call them supported models. We need to merge and optimize both sets of models from 01 and 02 and create the final set of supported models.

Consequences of supported models

Once we agree on the set of supported models, that needs to be supported by DMN editor, we need to use them as inputs for other test suites, especially for:

• scesim-editor test suite
• dmn-runner test suite
• dev-deployment test suite
• vscode-extension test suite (02 Reusing logic models only)
• online-editor test suite (02 Reusing logic models only)

Then wee need to keep in mind, that any change in supported models we produce as part of dmn-editor test suite may affect other test suites and all listed test suites should be rechecked and potentially synced with the dmn-editor test suite.

Supported environments

If we speak in the context of the dmn-editor test suite, we need probably to decide what is the supported environment for the test suite? Is it probably some matrix of the [browsers] x [operating system]?

Further restrictions

• We should never store DMN models from the users in the kie-tools repository, only if those models were sanitized, anonymized, and randomized.
• We should not exceed reasonable time for the dmn-editor test suite execution.

Key Features

• Putting nodes to the diagram, connecting them, and removing them
• Changing nodes properties, most important seems to be names and data types
• Creating custom data types

• ---

• Autolayout feature
• Keyboard accessibility
• Undo Redo operations
• Copy and Paste feature
• Bend points
• DRD
• Validation (e.g. nodes names)

If we have a look on the key features, it is highly recommended we organize the test into two phases:

• Execute crucial operation first, those above the delimiting line
• Execute nice to have features afterwards, those below the delimiting line

If we imagine test with steps:

• Create node A
• Create node B
• Undo
• Rename node A
• Paste node C
• Make connection A->B, A->C
• Add data type A
• Add data type B
• Move node C
• Add Bend point to the A->C edge
• Add constraint to data type A <--- and here the test start to fail
• Add B-> C edge
• Rename C node

The scenario above makes the test more similar to real user interaction, however makes almost impossible to maintain the test and analyze its results.

If the highlighted line with failure occurs, is that blocker for whole release? Is that some minor corner case bug? Hard to say as we mix a lot of different operations.

If we reorder the things according to the recommendation, and still testing the same model can be created:

• Create node A
• Create node B
• Paste node C
• Move node C
• Make connection A->B, A->C
• Add B-> C edge
• Add data type A
• Add data type B
• Rename node A
• Rename C node

• ---

• Undo
• Add Bend point to the A->C edge
• Add constraint to data type A  and here the test start to fail

This should be more readable and more effective form the testing time point of view as we do not switch the editor views so often.

Features that should be tested in other modules

• Modelling DMN nodes expressions - that should be tested as part of boxed-expression-component test suite
• Testing the DMN models using SceSim - that should be tested as part of scesim-editor test suite
• Executing the DMN models using DMN Runner - that should be tested as part of DMN Runner test suite
• Executing DMN models once they are deployed to cloud – that should be tested as part of the dev-deployment test suite
• Reusing models, as described in 02 should be somehow repeated, or let’s say sanity checked in online-editor and vscode-extension test suites
• Creating custom DMN types from the java files - that should be part of the vscode-extension test suite as it requires java files to be present. Those are supported only in the vscode-extension. The kie online-editor doesn't support java files. However this is currently blocked by ticket

[kbowers-ibm]

Thank you for creating such a thoughtful and detailed proposal Jozef, it looks great!

I agree that short test scenarios such as the ones you described are generally best avoided in such an e2e testing framework, because, as you mentioned, the functionality will be testing in more complex tests that will use that logic anyway. However the drawback then is that it could be harder to see why more complex tests are failing due to them having multiple steps and needing to debug in order to pinpoint where/why exactly it fails. The solution then as you said is to use unit tests or mocked tests, to perform these more basic tests, but should we then raise a separate ticket for the creation of such tests, or what do you think is best?

Regarding the where the models should be present for 01 Models from Scratch, and 02 Reusing Model, I think, for the sake of having a standardized practice everyone can follow, it makes the most sense to follow Luiz' example and have them inside the relevant package as part of some newly created storybook documentation.

Regarding the testing scenarios under Key Features, I think that it makes more sense to minimize the amount of times we switch views, and agree that option two would be the better test case.

[ljmotta]

Hey @jomarko , I've collected here all the functionalities of the DMN Editor I could think of, probably it's still missing a few things. I know we are still in the exploratory area, but I guess it can help us give north of what we will eventually cover on the tests.

functionalities

• autolayout
• boxed expression
• clipboard
• data types
  • creation
  • deletion
  • structure
  • collection
  • constraint
• diagram
  • (all mutations, we can check on the mutations folder) ⌛ https://github.com/apache/incubator-kie-tools/pull/2158
  • connections
  • drd
    • creation
    • node import
  • edges
  • nodes
  • selection
• draggable
  • test the component or the usage on constraint and documentation links?
• externalNodes
• feel
  • inline feel components
• includedModels
  • namespace
  • dependencies
  • label
  • pmml
• keyboard shortcuts
• overlaysPanel
• propertiesPanel
  • boxed expression
  • bkm node
  • decision node
  • input data node
  • knowledge source node
  • decision service node
  • group node
  • text annotation node
  • global
  • multiple nodes
  • unkown node
  • shape options
  • font options
  • documentation
  • diagram
• svg

non functional

• re-renders?
  • the store folder.