Slow Petri Net Unfolding (the Bumblebee Observations)

[Heizmann]

The Petri net unfolding is at the moment (early Oct 2019) the major bottleneck of the Petri Automizer verifier. The Petri net unfolding seems to be unnecessarily slow.

The following list of observations should help us to discuss improvements.

The Dinosaur Observations:

• D01: Typically, we have dozens of conditions per place.
• D02: Only few candidate for possible extension finally evolve to a possible extension
• D03: >90% CPU time is spend for checking if two conditions are in co-relation, but the implementation of this check is efficient

The Bumblebee Observations:

• B01: We instantiate conditions in candidates incrementally, approx 10% of the initial 1-condition instantiated candidates finally evolve to at least one possible extension.
• B02: While evolving a candidate we try to instantiate places with conditions and iterate over all conditions (dozens, see D01) that have a certain place.
• B03: Because of B02 the high cost are probably not due to the high number of the 1-condition instantiated candidates but due to the high costs for evolving a candidate.
• B04: Many instantiations fail because the condition is successor of a cut-off event.
• B05: The datastructures of the ConditionEventsCoRelation would allow us to get all conditions that are in co-relation to a given condition c. The number of all co-related conditions is typically high. Iterating over all of them is not efficient.
• B06: These datastructures would not allow us to get all conditions that are in co-relation to a given condition c and have a certain place p.
• B07: The B06 problem can be addressed: Do not store binary condition->event relation, but ternary condition->transition->event relation. Given condition c and place p, when asked for {c'| c' in co-relation to c and has place p} then find all predecessor transitions of p (separate tracking necessary, Petri nets only provide forward successors) get co-related event and filter successors for places.
• B08: We could save even more if we omit cut-offs from co-relation. Check with other users (backfolding, LBE) if this could become the default.
• B09: Memory optimization for internal data structure: We store for each condition the co-related events. Idea: Store the information only for the parent event (is a subset) and only the remaining diff for the condition. Saves probably also runtime because we have to copy fewer relations. Makes implementation more complex. (Probably wait until this becomes a bottleneck)
• B10: We could use co-relation information to reduce the number of 1-condition instantiated candidates in the first place. Orthogonal to B07. Requires additional callbacks for the on-demand construction. Makes implementation significantly more complex. Do not implement until this becomes the dominant bottleneck of the algorithm.

[Heizmann]

• B37: The computation of the AutomataOperationStatistics for the unfolding is costly (around 15%-20% additional costs for DifferencePairwiseOnDemand on the materialistic benchmark set). This computation is however rather harmless because it is done in the getAutomataOperationStatistics method. This method is called by the Automata Script Interpreter after executing the operation. The time is not added to the runtime of the operation (in the csv) but we will observe that the overall time taken by the test is longer than anticipated. (This looks like a bug, because if DifferencePairwiseOnDemand is not called directly getAutomataOperationStatistics will not be called and less time is spend.)

[Heizmann]

Here is now a related, but slightly different kind of analysis. While applying our difference operation to a Petri net and a finite automaton we do not only care about the speed of the algorithm but also about the size of the result. For the sake of simplicity I will measure the size of the Petri net by using the size of its flow relation. Furthermore, I will take the sum of all examples in the materialistic benchmark set although the numbers from lager examples may dominate over the numbers from smaller examples.

• Algorithm 1: DifferencePairwiseOnDemand (generation of statistics commented) Rumtime: 273s Size: 137121 CsvAutomataOperationStatistics1159606185_2020-03-24_06-42-53-853.csv.txt java.hprof.txt

• Algorithm 2: difference "PAIRWISE" Runtime 271s Size: 127558 CsvAutomataOperationStatistics1159606185_2020-03-24_06-53-05-556.csv.txt java.hprof.txt

• Algorithm 3: difference "PAIRWISE" with removal of redundant flow Runtime 484s Size: 101679 CsvAutomataOperationStatistics1159606185_2020-03-24_07-02-40-700.csv.txt java.hprof.txt

• Algorithm 4: difference "PAIRWISE" with removal of redundant flow and the mountain cock heuristic Runtime 465s Size: 102580 CsvAutomataOperationStatistics1159606185_2020-03-24_12-01-05-832.csv.txt java.hprof.txt

• Algorithm 5: difference "PAIRWISE" with removal of redundant flow for which we did a separate unfolding Runtime 884s Size: 101679 CsvAutomataOperationStatistics1159606185_2020-03-24_12-23-50-581.csv.txt java.hprof.txt

[Heizmann]

• B38: Based on the preceding evaluation
  1. We should use our algorithm for removing redundant flow.
  2. The main bottleneck is that RemoveRedundantFlow calls the isRestrictorCondition method very often and this method uses the co-relation of the unfolding.
  3. The moutain cock heuristic does speed up the time for RemoveRedundantFlow not save a lot of the overall time (at least on this benchmark set).
  4. It does make sense to reuse the existing unfolding for RemoveRedundantFlow.
  5. It is however strange, that an additional unfolding costs 400s although the initial unfolding which "drives" DifferencePairwiseOnDemand costs slightly less than 273s. My expectation was that vice versa the second unfolding is less costly than the first because the inputs are very similar but in the input of the second unfolding some redundant selfloops have already been removed.

[Heizmann]

• B39: 58d2c9e546c6350fc3f05447ed926566b9977491 might have a minor effect on the benchmark done two posts before (rumtime now slightly lower, size now slightly larger). I will however not redo the experiment because the bug does not spoil the conclusion in B39.

[Heizmann]

Here is now an experiment with the Svcomp20AutomizerConcurrentSpeedBenchmarks testsuite.

• Setting 1: No removal of redundant flow (469s) IncrementalLogWithBenchmarkResults_2020-03-26_21-58-18-352-incremental.log java.hprof.txt
• Setting 2: Removal of redundant flow within the construction of the difference (465s) IncrementalLogWithBenchmarkResults_2020-03-27_06-55-45-861-incremental.log java.hprof.txt
• Setting 3: Removal of redundant flow after the construction of the difference. (508s) IncrementalLogWithBenchmarkResults_2020-03-28_04-57-51-042-incremental.log java.hprof.txt
• Setting 4: Removal of redundant flow after the construction of the difference with backfolding (493s) IncrementalLogWithBenchmarkResults_2020-03-28_02-01-46-551-incremental.log java.hprof.txt
• Setting 5: Removal of redundant flow within the construction of the difference with backfolding (484s) IncrementalLogWithBenchmarkResults_2020-03-28_06-17-13-465-incremental.log java.hprof.txt
• Setting 6: Removal of redundant flow within the construction of the difference with backfolding and remerge of all places (==removal of dead transitions) (493s) IncrementalLogWithBenchmarkResults_2020-03-27_23-01-09-092-incremental.log java.hprof.txt

The interleaved.log shows that

• If the number of iterations is the same the setting with removal is slightly faster.
• The size of the Petri nets is smaller (with removal).
• The time of the emptiness check is lower (we can observe an effect of the reduced size in a place that is not affected by the extra costs of the removal operation.)
• The overall effect on the verification time is rather low.

The Hprof output shows that a major bottleneck of the verification algorithm will be that (as above) RemoveRedundantFlow calls the isRestrictorCondition method very often and this method uses the co-relation of the unfolding.

[Heizmann]

• B40: My statement in B37 was wrong. The time needed for the computation of statistics is added to the runtime in the CSV file. Hence, the validity of evaluations done in the past is questionable.

[Heizmann]

Here is another experiment with the Svcomp20AutomizerConcurrentSpeedBenchmarks testsuite, where I compared a version where counterexamples are computed during the difference operation with the version before.

• Additional unfolding for counterexamples. IncrementalLogWithBenchmarkResults_2020-04-04_14-52-43-739-incremental.log
• Counterexamples constructed via difference operation. IncrementalLogWithBenchmarkResults_2020-04-04_15-04-13-424-incremental.log

The interleaved.log shows.

• The time for emptiness checks is now indeed zero (i.e., there is no obvious bug)
• The second version is slightly faster on the benchmark set.
• Both versions are not really comparable. The random effect of different counterexamples is probably roughly as high as the effect of the time that we gain by saving an unfolding.

Nonetheless, the second version should become the default. (Already implemented by now)

[Heizmann]

I just rerun the materialistic benchmark set on 82f65192ab.

The comparison to benchmark results above hints that our Petri net unfolding became slower in the last two years. The size of the flow relation is however similar.

Algorithm 1: DifferencePairwiseOnDemand (generation of statistics commented) Rumtime: 317s Size: 137121 CsvAutomataOperationStatistics903064416_2022-08-19_18-52-02-170.csv.txt

Algorithm 2: difference "PAIRWISE" Runtime 306s Size: 127558 CsvAutomataOperationStatistics903064416_2022-08-19_18-45-14-078.csv.txt

[Heizmann]

I just rerun the materialistic benchmark set on https://github.com/ultimate-pa/ultimate/commit/b48bd9feeee0f1bfc84257002261e60595ee34c1

The Petri net unfolding became around 35% slower since https://github.com/ultimate-pa/ultimate/commit/82f65192ab58bd843b52489f1859a782c12bfc44

Algorithm 1: DifferencePairwiseOnDemand (generation of statistics commented) Rumtime: 431s Size: 137121 CsvAutomataOperationStatistics903064416_2022-08-25_17-05-42-503.csv

Algorithm 2: difference "PAIRWISE" Runtime 424s Size: 127558 CsvAutomataOperationStatistics17331878_2022-08-25_17-32-09-423.csv