Cornutum / tcases

A model-based test case generator
MIT License
218 stars 54 forks source link

Unable to resolve a value after 10000 tries #276

Closed finndfr closed 1 year ago

finndfr commented 1 year ago

I am hitting the following error when trying to create an input type that is an array of enums. I actually have this structure in my input several times, but I'm still having the same error with this reduced example. Maybe this is a bug? Or maybe I'm missing something in my definition?

org.cornutum.tcases.resolve.ResolverSkipException: Unable to resolve a value after 10000 tries
    at org.cornutum.tcases.resolve.ResolverContext.tryUntil(ResolverContext.java:111)
    at org.cornutum.tcases.resolve.ResolverContext.tryUntil(ResolverContext.java:88)
    at org.cornutum.tcases.resolve.ArrayDomain.lambda$getNextItem$9(ArrayDomain.java:248)
    at org.cornutum.tcases.util.ExecutionContext.resultFor(ExecutionContext.java:40)
    at org.cornutum.tcases.resolve.ArrayDomain.getNextItem(ArrayDomain.java:246)
    at org.cornutum.tcases.resolve.ArrayDomain.newArray(ArrayDomain.java:218)
    at org.cornutum.tcases.resolve.ArrayDomain.lambda$values$3(ArrayDomain.java:178)
    at java.util.stream.StreamSpliterators$InfiniteSupplyingSpliterator$OfRef.tryAdvance(StreamSpliterators.java:1361)
    at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
    at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
...

My simplified input file is as follows. I am invoking with a plain tcases <project>.

{
  "system": "Fruit",
  "order": {
    "food": {
      "fruit": {
        "type": "array",
        "minItems": 0,
        "maxItems": 4,
        "uniqueItems": true,
        "items": {
          "enum": [
            "apple",
            "banana",
            "grape",
            "strawberry"
          ]
        }
      }
    }
  }
}

My log:

15:38:53.875 INFO  org.cornutum.tcases.Tcases - Tcases 4.0.1 (2023-01-16)
15:38:53.914 INFO  org.cornutum.tcases.Tcases - Reading system input definition from ./fruit-Input.json
15:38:54.170 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Preparing constraint info
15:38:54.173 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Generating test cases
15:38:54.177 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Extended 0 valid base test cases
15:38:54.186 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Created 2 valid test cases
15:38:54.186 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Extended 0 base failure test cases
15:38:54.187 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Created 3 failure test cases
15:38:54.187 INFO  o.c.t.generator.TupleGenerator - FunctionInputDef[order]: Completed 5 test cases
kerrykimbrough commented 1 year ago

Thanks for your helpful description of this problem. Yes, it is a Tcases defect. A fix is coming soon.

Here's what's happening. Per standard procedure, Tcases generates a failure case with an array value larger than maxItems. But that can't be done without violating the uniqueItems condition. Hence the "Unable to resolve" error.

The first problem is that this error should just skip the test case and proceed, not bring the whole program to a halt. The next problem is that the message offers no explanation for what went wrong. But the root problem is that the infeasibility of this case should have been recognized earlier -- no such failure should have been generated at all.

finndfr commented 1 year ago

Thanks for the quick reply. My real input is an unbounded list of enum strings, with an enum cardinality of 4. But I added both the uniqueItems and maxItems constraint to try to make things simpler. As a workaround I've had a bit more luck modeling this as 4 booleans such as contains_banana, assuming duplicates are dealt with.

kerrykimbrough commented 1 year ago

Fixed in release 4.0.2