Inconsistency between compilations

[X-C3LL]

Hi!

I was testing the project with this simple grammar rules:

{
    "<START>": [
        ["<?php", "<FUZZ>", "\n?>"]
    ],
    "<FUZZ>": [
        ["\ntry {\n try {\n", "<DEFCLASS>", "\n} catch (Exception $e){}} catch(Error $e){}", "<FUZZ>"],[]
    ],
    "<DEFCLASS>" : [
        ["class ", "<CLASSNAME>", " {\n\t", "<CLASSBODY>","}\n"],[]
    ],
    "<CLASSNAME>" : [
        ["Class01"],
        ["Class02"],
        ["Class03"],
        ["Class04"],
        ["Class05"]
    ],
    "<CLASSBODY>" : [
        ["test01;\n"],
        ["test02;\n"],
        ["test03;\n"]
    ]
}

Every time I compile (make -j$(nproc) GRAMMAR_FILE=grammars/phpexcept.json) and test the rules with the generator I obtain different results: sometimes it only picks a rule, other a small chain of rules...:

 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   ./grammar_generator-phpexcept 100 1000 ./seeds ./trees
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/1
class Class03 {
    test01;
}

psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   make clean && make -j$(nproc) GRAMMAR_FILE=grammars/phpexcept.json && ./grammar_generator-phpexcept 100 1000 ./seeds ./trees
psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/1
<?php
try {
 try {
class Class04 {
    test03;

}

} catch (Exception $e){}} catch(Error $e){}
?>

psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   make clean && make -j$(nproc) GRAMMAR_FILE=grammars/phpexcept.json && ./grammar_generator-phpexcept 100 1000 ./seeds ./trees
psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/1
Class01

I am not sure if I am creating the rules in a wrong way (but checking documentation and the ruby example it looks fine to me).

[h1994st]

Hi @X-C3LL ,

Your grammar looks fine. the generated test cases are good as well, except for your last example. The generator does produce different results every time due to randomness. You may want to set a fixed random seed, as shown in the last argument of the command below, for the grammar generator.

May I know your expected outputs? I tried your grammar on my side, using the latest dev branch and the following command. The generator works as I expected.

./grammar_generator-phpexcept 100 1000 ./seeds ./trees 1641180035

[X-C3LL]

Hi!

Setting your default value:

 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   ./grammar_generator-phpexcept 100 1000 ./seeds ./trees 1641180035
Using seed 100
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/*                                                      
Class05Class03Class04Class04Class05Class03Class05Class03Class05Class02Class04Class05Class03Class05Class04Class02Class03Class01Class01Class05Class02Class05Class03Class01Class03Class01Class04Class03Class01Class01Class05Class05Class04Class05Class03Class04Class02Class03Class01Class03Class01Class02Class02Class05Class02Class05Class01Class04Class05Class04Class01Class05Class02Class03Class01Class04Class02Class02Class02Class01Class04Class03Class04Class03Class02Class05Class04Class01Class05Class04Class05Class03Class01Class02Class05Class01Class02Class02Class05Class04Class05Class05Class05Class04Class05Class05Class01Class05Class04Class05Class05Class04Class05Class03Class03Class03Class01Class02Class03Class02

If I change the fixed value, the result it's the same:

 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   sudo rm seeds/* && sudo rm trees/*                               
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   ./grammar_generator-phpexcept 100 1000 ./seeds ./trees 1337180035
Using seed 100
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/*                                                  
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   ./grammar_generator-phpexcept 100 1000 ./seeds ./trees 1337181337
Using seed 100
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/*
Class05Class03Class04Class04Class05Class03Class05Class03Class05Class02Class04Class05Class03Class05Class04Class02Class03Class01Class01Class05Class02Class05Class03Class01Class03Class01Class04Class03Class01Class01Class05Class05Class04Class05Class03Class04Class02Class03Class01Class03Class01Class02Class02Class05Class02Class05Class01Class04Class05Class04Class01Class05Class02Class03Class01Class04Class02Class02Class02Class01Class04Class03Class04Class03Class02Class05Class04Class01Class05Class04Class05Class03Class01Class02Class05Class01Class02Class02Class05Class04Class05Class05Class05Class04Class05Class05Class01Class05Class04Class05Class05Class04Class05Class03Class03Class03Class01Class02Class03Class02%

Looks like it is taking the number of seeds (first arg) as parameter instead of the last arg (Just saw your last commit). But even changing the fixed value, all the seeds generated contains only the rule <CLASSNAME>. If I recompile and try the same seed value, it picks another rule or a small chain of rules, but rarely applies all of them.

What I was expecting is generating a batch of variants of the next example (or the same with recursion at <FUZZ> level) where all the rules were applied:

<?php
try {
 try {
class Class04 {
    test03;

}

} catch (Exception $e){}} catch(Error $e){}
?>

The PHP logic does not matter, I was using a bigger file with all the rules, just created this small grammar to illustrate the issue

---

OTOH I had to edit grammars/f1_c_gen.py:

 import sys
 import itertools
 import random
 import os
 import string
 import json

 from f1_common import LimitFuzzer

 class TreeNode:
     #node_type: int = 0
     #rule_id: int = 0
     #val: str = ''
     #parent: 'TreeNode' = None
     #subnodes: list = None

I had to comment it to avoid this error:

  File "grammars/f1_c_gen.py", line 39
    node_type: int = 0
             ^
SyntaxError: invalid syntax
  File "grammars/f1_c_gen.py", line 39
    node_type: int = 0
             ^
SyntaxError: invalid syntax

[h1994st]

~Looks like it is taking the number of seeds (first arg) as parameter instead of the last arg~ (Just saw your last commit). But even changing the fixed value, all the seeds generated contains only the rule <CLASSNAME>. If I recompile and try the same seed value, it picks another rule or a small chain of rules, but rarely applies all of them.

What I was expecting is generating a batch of variants of the next example (or the same with recursion at <FUZZ> level) where all the rules were applied

That is weird. I did generate test cases with recursive <FUZZ> rule, like:

<?php
try {
 try {
class Class04 {
        test01;
}

} catch (Exception $e){}} catch(Error $e){}
try {
 try {

} catch (Exception $e){}} catch(Error $e){}
try {
 try {
class Class05 {
        test02;
}

} catch (Exception $e){}} catch(Error $e){}
try {
 try {

} catch (Exception $e){}} catch(Error $e){}
?>

  File "grammars/f1_c_gen.py", line 39
    node_type: int = 0
             ^
SyntaxError: invalid syntax
  File "grammars/f1_c_gen.py", line 39
    node_type: int = 0
             ^
SyntaxError: invalid syntax

This error did not show up as well. May I know your python3 version?

[X-C3LL]

Hi

My version:

Python 3.7.3

[h1994st]

Thanks. The python version looks fine.

One possible reason for your issue (no recursive <FUZZ> rule) might be that <CLASSNAME> is wrongly identified as the entry rule for the grammar. Can you share with me the generated f1_c_fuzz.h in include directory?

[X-C3LL]

Hi!

I tested it on two machines with a more recent version of my distro (Debian 10) and it works like a charm (just needed to do the .py edit again):

psyconauta@insulaanglia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/6
<?php
try {
 try {
class Class02 {
    test02;
}

} catch (Exception $e){}} catch(Error $e){}
try {
 try {
class Class04 {
    test03;
}

} catch (Exception $e){}} catch(Error $e){}
?>%                                          

So it's highly probable the problem is in my side. Let me upgrade the VM (currently is running Debian 9) and test it to confirm it.

[X-C3LL]

Yep, the problem was on my side. Upgrading to Debian 10 solved the issue: every time the whole chain of rules is used.

 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   lsb_release -d                                         
Description:    Debian GNU/Linux 10 (buster)
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   ./grammar_generator-phpexcept 100 1000 ./seeds ./trees 
Using seed 1641239392
 psyconauta@insulaalchimia ᐓ  ~/Grammar-Mutator |stable⚡ ᐓ   cat seeds/1
<?php
try {
 try {
class Class01 {
    test03;
}

} catch (Exception $e){}} catch(Error $e){}
try {
 try {
class Class05 {
    test02;
}

} catch (Exception $e){}} catch(Error $e){}
try {
 try {

} catch (Exception $e){}} catch(Error $e){}
try {
 try {

} catch (Exception $e){}} catch(Error $e){}
?>%