search

citrusframework / citrus-simulator

Standalone simulator for different messaging transports such as Http REST, SOAP WebService, JMS, RMI, mail messaging and more
Apache License 2.0
44 stars 45 forks source link

problem with in-memory pagination when having high volume of data #255

Closed bbortt closed 4 months ago

bbortt commented 5 months ago

Bug

we're running a central instance of a simulator, which contains around 30'000 scenario executions right now. whenever we access the "Scenario Execution" UI (/scenario-result), the simulator instantly runs out of memory.

Expected Behavior

The UI displays the paginated Scenario Execution information (without problems).

More Detailed Explanation

We're in the ScenarioExecutionQueryService, about to fire the query. The simulator logs a warning in the moment the query will be executed:

[TIMESTAMP]: HHH90003004: firstResult/maxResults specified with collection fetch; applying in memory

The problem is, that because of JOIN FETCH clauses, the SQL query does not include the paging information. That is not possible, due to the nature of the result. See this good blog by Thorben Janssen about the problem.

What should be done instead, is selecting the ID's of the entities using a first query with pagination, then selecting the full entity with relationship in a second query - limited to the previously selected ID's.

bbortt commented 4 months ago

note that only methods with the signature method(Specification, Pageable) that are annotated with an @EntityGraph are affected by the problem.