codelion / gramtest

GramTest: A tool for Grammar based Test case generation
Apache License 2.0
85 stars 17 forks source link

Low quality of generated outputs #8

Closed jovanpop-msft closed 7 years ago

jovanpop-msft commented 7 years ago

In some cases outputs generates the same values for difference occurrences of literals. It is not critical, but it would be better if we have different values.

As an example, the following call should generate JSON outputs:

java -jar gramtest-0.2.1.jar -file json.bnf -tests MYTEST -max 100 -min 10

String/number literals in the most of the outputs have the same values Z and 9: {"Z":{"Z":[9,{"":,"":}],"Z":[9,{"":,"":}]},"Z":{"Z":[{"":,"":},{"":,"":}],"Z":[9,{"":,"":}]}}

It is strange because in some other cases I'm getting better outputs.

Note that I'm using modified grammar described in #7

codelion commented 7 years ago

I suspect the main issue is that the generator tends to go down in a depth-first fashion while generating test cases. This means it may miss the alternative paths in the grammar when it finds a recursive production and keep going down the recursion until it hits max depth.

The best way to handle it would be to force choice of a different branch. This may be done by using a breadth-first strategy when max depth is hit. I will need to think a bit more before implementing a fix.

codelion commented 7 years ago

This is now fixed in the master. For the same grammar it now generates the following:

{"JLgeXHHT":{"vlkHt":91962,"Cfhfv":87163},"qgkMnsUQ":[499884,true]}
{"JLgeXHHT":{"vlkHt":91962,"Cfhfh":87186},"qgkMnsUV":[[{"C":6,"D":4},9846],false]}
{"JLgeXHHx":{"vlkHM":91945,"Cfhfh":87186},"qgkMnsUV":[499803,219514]}
{"JLgeXHHx":{"vlkHM":91945,"CfhfV":87189},"qgkMnsUM":[499807,219514]}