camunda-community-hub / bpmn-driven-testing

Visually select paths through a BPMN process as test cases. Generate and enrich those test cases for easier unit testing of your process implementations.
Apache License 2.0
30 stars 10 forks source link

Testing Multi-Instance with Collection #192

Closed bennobuilder closed 1 week ago

bennobuilder commented 2 weeks ago

I’m currently working on a test case using BPMN-Driven-Testing and encountered an issue while trying to test a multi-instance subprocess configured with a collection (camunda:collection, see image).

In my test, I expect the verifyLoopCount to reflect the size of the collection (in this case, 2 for a list of two emails), but it always returns 0. The variables are being consumed (-> to verify ProcessInstance contains variable emails but empty).

Is there support for testing camunda:collection in this context (couldn't find any test) or is only loop cardinality supported. An example or guidance would be appreciated.

Thanks :)

image

@SpringBootTest(classes = {Application.class, BpmndtConfiguration.class})
class EmailBounceHandlingProcessTest {

  @RegisterExtension
  TC_EmailBounceHandlingProcess_TC1 tc1 = new TC_EmailBounceHandlingProcess_TC1();

  @Test
  void testTC1() {
    var emailsVariable = List.of("email1.eml", "email2.eml");

    tc1.handleCallProcessEveryMessageSubprocess().verifyParallel();
    // TODO: Can we test "camunda:collection" with BPMN-Driven-Testing
    // tc1.handleCallProcessEveryMessageSubprocess().verifyLoopCount(emailsVariable.size());
//    tc1.handleCallProcessEveryMessageSubprocess().handle(0).verify((pi, callActivity) -> {
//      pi.variables().containsKey("email");
//    });
//    tc1.handleCallProcessEveryMessageSubprocess().handle(1).verify((pi, callActivity) -> {
//      pi.variables().containsKey("email");
//    });

    tc1.createExecutor()
       .withBean("collectAllEmails", new DelegateMock((new HashMap<String, Object>() {{
         put("emails", emailsVariable);
       }})))
       .verify(pi -> {
         pi.variables()
           .containsKey("emails"); // empty list
         pi.isEnded();
       })
       .execute();
  }
}
gclaussn commented 2 weeks ago

hi @bennobuilder,

on branch issue/192 (see commit) I added an example process with a multi instance call activity that uses a collection expression ${elements} and element variable element. A test starts a test case with variable elements=["a", "b", "c"] and verifies the loop count as well as the variables at the call activity within the multi instance.

it is a bit tricky to test the local element variable, used within the multi instance.

in the verify method of the call activity handler of the multi instance a ProcessInstanceAssert (here named piAssert) is provided. but it can only be used to assert variables that are on process instance level, e.g. elements. to verify local variables, like the element variable, the local variables of the related exectution must be retrieved. to do this, the execution ID is required. therefore the historic activity instance of the multi instance body must be found.

    tc.handleCallActivity().handle(0).verify((piAssert, callActivityDefinition) -> {
        piAssert.variables().containsKey("elements"); // assertion of process scoped variables only (e.g. collection)

        // find process instance of test case, e.g. by business key
        var pi = tc.getProcessEngine().getRuntimeService().createProcessInstanceQuery()
            .processInstanceBusinessKey("bk")
            .singleResult();

        // find historic activity instance of call activity in multi instance scope
        var historicActivityInstance = tc.getProcessEngine().getHistoryService().createHistoricActivityInstanceQuery()
              .activityId("multiInstanceCallActivity#multiInstanceBody") // BPMN element ID + #multiInstanceBody
              .singleResult();

        // get local variables (including element variable of collection) of related execution
        var localVariables = tc.getProcessEngine().getRuntimeService().getVariablesLocal(historicActivityInstance.getExecutionId());
        assertThat(localVariables.get("element"), equalTo("a"));
    });

when the element variable is propagated to the subprocess via delegate variable mapping, in mapping propagation or in mappings, the variable value is much easier to assert.

    tc.handleCallActivity().handle(0).verifyInput(variables -> {
        assertThat(variables.getVariable("element"), equalTo("a"));
    });
    tc.handleCallActivity().handle(1).verifyInput(variables -> {
        assertThat(variables.getVariable("element"), equalTo("b"));
    });
    tc.handleCallActivity().handle(2).verifyInput(variables -> {
        assertThat(variables.getVariable("element"), equalTo("c"));
    });

this example shows that a multi instance call activity with collection expression is supported.

regarding your test, I'm not sure what DelegateMock is doing. maybe the emails variable is not set correctly.

i hope the example helps. if you have more questions, feel free to ask.

best regards!

bennobuilder commented 2 weeks ago

Hi @gclaussn,

Thanks for the detailed explanation and example. After further investigation, I identified two issues that contributed to my failing tests.

Discovery 1: Variable Overriding by Output

I found that defining Outputs in a Delegate expression can override initially set variables (e.g., by tc1.createExecutor().withVariable(..)). This caused the issue I initially encountered, where I expected variables to persist but found they were overwritten by an Output (without values) defined in the modeler (see Screenshot 1).

Screenshot 1 ![image](https://github.com/user-attachments/assets/a6b859a4-1a67-4446-a292-7f90a5da3168)

Discovery 2: Different Behavior with Spring Boot

The bpmn-driven-testing library behaves differently when integrated with Spring Boot compared to a plain Java setup.

With Spring Boot

Running multiple test cases simultaneously results in exceptions like OptimisticLockingException and ProcessEngineException (see Screenshot 2):

org.camunda.bpm.engine.ProcessEngineException: ENGINE-14026 No job found with id 'a35e4042-66c8-11ef-a37f-00059a3c7a00'

org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of 'DELETE MessageEntity[b9969346-66c8-11ef-9cb8-00059a3c7a00]' failed. Entity was updated by another transaction concurrently.
Screenshot 2 ![image](https://github.com/user-attachments/assets/c3576d87-52b3-45ff-a506-3a14b961f7d8)

Running only the first test case (similar to my initial test case but simplified, see Test Case) results in an AssertionError (see Screenshot 3):

java.lang.AssertionError: Expected at least one job for activity 'Call_Participant_2'
Screenshot 3 ![image](https://github.com/user-attachments/assets/365577d5-9f8d-4124-98d1-eca7a2cff092)

Without Spring Boot

When I remove Spring Boot (i.e., the SpringBootTest annotation), the tests run as expected without exceptions (see Screenshot 4). I confirmed this by setting the expected loop count to 2 and anticipating an error.

Screenshot 4 ![image](https://github.com/user-attachments/assets/c63515d2-0464-46c8-b4a1-648d5ef662ed)

Test Case

These issues were observed using the following files, added to the advanced-spring-boot integration tests (checked out locally on Windows 11 with Java 21):

participants.bpmn ```xml Participant1_TC1 StartEvent_Participant_1 CollectElementsTask Call_Participant_2 EndEvent_Participant_1 Flow_19m8hkc 0 0 1 1-31 1-12 ? Flow_1hqadxe Flow_0q1lqbh Flow_1hqadxe Flow_19m8hkc Flow_0q1lqbh Participant3_TC1 StartEvent_Participant_3 UserTaks_1 EndEvent_Participant_3 Flow_08dpap9 Flow_08dpap9 Flow_00eq203 Flow_00eq203 Participant2_TC1 StartEvent_Participant_2 Call_Participant_3 EndEvent_Participant_2 Flow_0i6rvwd Flow_0i6rvwd Flow_0rqjk1w Flow_0rqjk1w ```
ParticipantsTest.java ```java package org.example.it; import generated.participant_1_pid.TC_Participant1_TC1; import generated.participant_2_pid.TC_Participant2_TC1; import org.camunda.bpm.engine.test.assertions.bpmn.ProcessInstanceAssert; import org.example.ExampleApp; import org.example.it.utils.DelegateMock; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.RegisterExtension; import org.springframework.boot.test.context.SpringBootTest; import java.util.List; @SpringBootTest(classes = {ExampleApp.class, BpmndtConfiguration.class}, webEnvironment = SpringBootTest.WebEnvironment.NONE) public class ParticipantsTest { @RegisterExtension public TC_Participant1_TC1 tc1 = new TC_Participant1_TC1(); @RegisterExtension public TC_Participant2_TC1 tc2 = new TC_Participant2_TC1(); @Test public void testExecuteP1TC1() { tc1.handleCall_Participant_2().verifyParallel(); tc1.handleCall_Participant_2().verifyLoopCount(3); tc1.createExecutor() // .withBusinessKey("bk") .withVariable("elements", List.of("a", "b", "c")) .withBean("collectElements", new DelegateMock()) .verify(ProcessInstanceAssert::isEnded) .execute(); } @Test public void testExecuteP2TC1() { tc2.handleCall_Participant_3().verify((piAssert, callActivityDefinition) -> { }); tc2.createExecutor() // .withBusinessKey("bk") .verify(ProcessInstanceAssert::isEnded) .execute(); } } ```

Is there a known issue with the bpmn-driven-testing library in a Spring Boot environment? Do you recommend using Spring Boot for these tests, or would sticking with plain Java be preferable? We chose Spring Boot because our project uses it but for the tests we would also be fine using plain Java, I guess.

Thanks :) and best regards

gclaussn commented 1 week ago

hi @bennobuilder,

I think I know what the problem with your Spring Boot tests might be.

the advanced-spring-boot integration test does not look like a project that I have usually under test - since it tests only the configration of the process engine, provided by the camunda-bpm-spring-boot-starter, when exposing the BpmndtProcessEnginePlugin as a Spring bean.

when running process tests (with or without bpmn-driven-testing), the application.yaml under src/test/resources must disable the job executor via:

camunda:
  bpm:
    job-execution:
      enabled: false

(see https://docs.camunda.org/manual/latest/user-guide/spring-boot-integration/configuration/#camunda-engine-properties)

by default the job executor is enabled and runs in parallel in a separate thread. this makes the process tests not deterministic and leads to the exceptions you described (No job found or Entity was updated by another transaction concurrently). therefore it must be disabled while testing. I will add this configuration to the Spring Boot integration test to make it complete.

please give me a quick feedback if this makes your Spring Boot based tests run successfully.

best regards

bennobuilder commented 1 week ago

Hi @gclaussn,

Thanks for your suggestion. I tested it in a demo project, and it works perfectly. However, for our main project, we decided to proceed without Spring Boot testing for now since it's easier to setup and doesn't seem to be necessary to test the BPMN, even though the project itself uses Spring Boot. Your library has made the overall testing process much smoother.

Thanks again, and cheers :)