How to improve Zest fuzzing on Spring API

[llaumegui27]

Hello I try to do some fuzzing test on my Java Spring API. The goal is to test the createPerson() function which create a person in my database (curl request example : curl -X POST -H "Content-Type: application/json" -d '{"name":"Doe", "firstname":"John", "birthDate":"2001-09-21", "phone":"0123456789", "mail":"john.doe@mail.com"}' http://localhost:8080/api/persons

For this I used the "fuzzing a compiler" tutorial. I wrote a fuzzing class test which test the createPerson function and a generator class which generate random data like random string for name, firstname, phone and mail and a random date for birthdate.

And before to start my server and launch my test, my first question is : Is JQF can make API request, it can fuzz an API ? So, I managed to launch my test but 0 bug found, 0.03% coverage with around 17 branches -_-. At this point I don't think any requests have been sent to the API. So I created a server mock with mockito to see if the problem came from contacting the server even if it's local. And I've this results :

Test name:            com.example.demo.controller.PersonControllerTest#testCreatePerson
Results directory:    /full/path/to/api/javafuzzapi/target/fuzz-results/com.example.demo.controller.PersonControllerTest/testCreatePerson
Elapsed time:         30m 0s (max 30m 0s)
Number of executions: 1 328 912 (no trial limit)
Valid inputs:         1 328 912 (100,00%)
Cycles completed:     1097
Unique failures:      0
Queue size:           8 (2 favored last cycle)
Current parent input: 0 (favored) {159/980 mutations}
Execution speed:      772/sec now | 738/sec overall
Total coverage:       4 530 branches (6,91% of map)
Valid coverage:       4 530 branches (6,91% of map)

So a big gap between 0.03% and 6.91% how you can explain this ? The mock server probably but why ? Even if I have a big increase of coverage I don't know if it is correct or not I don't know how coverage is calculated can you explain me how coverage is calculated ?

I still think that 7% is very little and there are still no bugs found. If anyone can answer my questions that would be great. Tell me If you need to the more like the source to help me.

[rohanpadhye]

Two points:

1. The 6.91% number you are seeing is a percentage of the coverage map, not a percentage of your code. JQF does not know how big your code is because it instruments everything including libraries, etc. and it does not know what total amount of code is reachable from the fuzz entry point. So, only focus on the absolute number (4530 branches), which looks quite reasonable. There is no way for JQF to give you a percentage of total coverage unless you know how many total branches are theoretically executable from your entry point (an undecidable problem in general).

2. If you are calling an HTTP API endpoint via a network call, then JQF will not be able to collect coverage of the API endpoint. JQF will only see the method call that makes the network request. If you want to fuzz the server, you need a test method that direclty calls to API endpoint handler (e.g., handleGET or similar) and ask JQF to generate the request data as a JSON object (or better yet, of the exact type that internal functions process after unpacking the JSON object in the regular server, such as class Person).

[llaumegui27]

Hello @rohanpadhye, thx for your answer. I tried something but It doesn't works and I don't understand why because I was inspired by the example of "fuzzing a compiler", this one :

@Fuzz
    public void testWithString(String code) {
        SourceFile input = SourceFile.fromCode("input", code);
        compile(input); // No assertions; we are looking for unexpected exceptions
    }

My error when I run mvn jqf:fuzz -Dclass=com.example.demo.controller.PersonControllerFuzz -Dmethod=testCreatePerson -Dtime=5m:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.051 s
[INFO] Finished at: 2024-07-17T14:38:36+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal edu.berkeley.cs.jqf:jqf-maven-plugin:1.8:fuzz (default-cli) on project javafuzzapi: Fuzzing resulted in the test failing on 1 input(s). Possible bugs found. Use mvn jqf:repro to reproduce failing test cases from /full/path/to/api/javafuzzapi/target/fuzz-results/com.example.demo.controller.PersonControllerFuzz/testCreatePerson/failures. Sample exception included with this message.: Cannot find generator for com.example.demo.controller.PersonControllerFuzz.testCreatePerson:person of type com.example.demo.model.Person -> [Help 1]

My fuzzing test class :

import [...]

@RunWith(JQF.class)
public class PersonControllerFuzz {

    PersonGenerator generator = new PersonGenerator();
    SourceOfRandomness random = new SourceOfRandomness(new Random());

    @Autowired
    private RestTemplateBuilder restTemplateBuilder;
    private PersonController personController;

    static {
        // Disable all logging by Closure passes, to speed up fuzzing
        java.util.logging.LogManager.getLogManager().reset();
    }
    @Fuzz
    public void testCreatePerson(Person person) {

        RestTemplate restTemplate = restTemplateBuilder.build();
        String url = "http://localhost:8080/api/persons";

        // Creer une nouvelle personne
        person.setNom(generator.randomString(random));
        person.setPrenom(generator.randomString(random));
        person.setDateNaissance(generator.randomDate(random));
        person.setNumTelephone(generator.randomString(random));
        person.setMail(generator.randomString(random));

        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);

        HttpEntity<Person> request = new HttpEntity<>(person, headers);

        Person response = restTemplate.postForObject(url, request, Person.class);
    }
}

For info, PersonGenerator is a class in the same directory which contains two functions which return a random string and a random local date. Then I made this test without annotation (without JQF) to see if it works and it works well, injected in my API : :

{
    "id": 27,
    "nom": "?]xëTjNà|é1I",
    "prenom": "><:Pê<_f<ùûl@B/k'ryöRqdQGMi`4cùHWPCqcXz9öQBTA4DêêI_,0ù|x²;UjWmD`²RycCW~8>9TsgöL",
    "dateNaissance": "2723-01-05",
    "numTelephone": "%[µéâ_[n-aê²WEiAF!)Y^^).D",
    "mail": "}aclùQZq1â<M4kY~}-à|£Kcy!mµ,yB!§&nµ+²y>i##²°p<;r7UäEX%|k^0i|°S]))`&.<!"
  }

Maybe missing something to adapt my test and launch it with JQF but why ? idk

[rohanpadhye]

You're using the concept of generators incorrectly. You don't have to create an instance of generator and call methods like randomString inside the test. The idea is that the generator is used to construct a randomly generated value (in your case, a Person object) and provide it as a test argument. Please go back to the tutorial and see how this is done for the compiler. In your case, the test would like something like this:

    @Fuzz
    public void testCreatePerson(@From(PersonGenerator.class) Person person) {

        RestTemplate restTemplate = restTemplateBuilder.build();
        String url = "http://localhost:8080/api/persons";

        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);

        HttpEntity<Person> request = new HttpEntity<>(person, headers);

        Person response = restTemplate.postForObject(url, request, Person.class);
    }

Where PersonGenerator extends Generator<Person> to provide random person objects:

public class PersonGenerator extends Generator<Person> {
  public void generate(SourceOfRandomness random, GenerationStatus status) {
       person = new Person();

        // Creer une nouvelle personne
        person.setNom( ... random string ... );
        ....

        return person;

  }
}

For full reference, see the documentation on generators at the JUnit-Quickcheck project, which JQF is based on: https://pholser.github.io/junit-quickcheck/site/1.0/usage/other-types.html

[llaumegui27]

Yeah I tried with this setup but I thought I was on the wrong track, so I wanted to try it without a generator. In this case the test launch and stopped :

Test name:            com.example.demo.controller.PersonControllerFuzz#testCreatePerson
Results directory:    /full/path/to/api/javafuzzapi/target/fuzz-results/com.example.demo.controller.PersonControllerFuzz/testCreatePerson
Elapsed time:         14s (max 5m 0s)
Number of executions: 98 730 (no trial limit)
Valid inputs:         0 (0,00%)
Cycles completed:     0
Unique failures:      2
Queue size:           0 (0 favored last cycle)
Current parent input: <seed>
Execution speed:      7 780/sec now | 6 938/sec overall
Total coverage:       0 branches (0,00% of map)
Valid coverage:       0 branches (0,00% of map)
Fuzzing stopped due to guidance exception: Too many trials without coverage; likely all assumption violations

Stopped by the 0 branches and 0% coverage a priori.

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.004 s
[INFO] Finished at: 2024-07-17T15:36:18+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal edu.berkeley.cs.jqf:jqf-maven-plugin:1.8:fuzz (default-cli) on project javafuzzapi: Internal error: Too many trials without coverage; likely all assumption violations -> [Help 1]

So 2 unique failures, mvn jqf:repro -Dclass=com.example.demo.controller.PersonControllerFuzz -Dmethod=testCreatePerson -Dinput=target/fuzz-results/com.example.demo.controller.PersonControllerFuzz/testCreatePerson/failures/id_000000 :

.id_000000 ::= FAILURE (java.lang.NullPointerException)
E
Time: 0,036
There was 1 failure:
1) testCreatePerson(com.example.demo.controller.PersonControllerFuzz)
java.lang.NullPointerException
        at com.example.demo.controller.PersonControllerFuzz.testCreatePerson(PersonControllerFuzz.java)

FAILURES!!!
Tests run: 1,  Failures: 1

(This is the same bug for both) And I think this bug happens because the test is not well written but I can't say what's wrong

[rohanpadhye]

You are on the right track; however, I'm sorry I can't help you debug your code step by step as I don't have the time resources to do so. For most people starting out with JQF but having no background in fuzzing, I would say try to run the example repo: https://github.com/rohanpadhye/jqf-zest-example. It should work as described (let me know if doesn't). Then, once you have a working repo, make small changes to get from there to your application. That way, you can identify the smallest change that caused something to break, and it will help you debug.

You can also use the many examples from the JQF-examples directory. If you want to write custom generators, the JGraphT tests are useful as they have many graph generators and generator-based fuzz tests.

Good luck!

[llaumegui27]

Thanks, but reproduce my app from this repository would take far too long. I think the person object in testCreatePerson is null but why I cant know because it's impossible to debug in JQF do you have just some tips to debug in JQF, if is it possible to use assert or something ?

[rohanpadhye]

Yes, you can debug JQF as it is just a Java program. With the Maven plugin can use -Dquiet to suppress JQF's status screen so that you can use println() to log whatever you need in generators or test methods. You can also run JQF programmatically (e.g., see Zest's integration tests) and then use your IDE's debugger to step-through the whole execution. Hope that helps.

[llaumegui27]

Ok I finally succeeded ! The NullPointerException error from restTemplate because the value of restTemplate was :

 RestTemplate restTemplate = restTemplateBuilder.build();

And apparently we need this Annotation to use restTemplateBuilder :

@RunWith(SpringRunner.class)
@SpringBootTest
public class PersonControllerFuzz ...

But we can't put two @RunWith (@RunWith(JQF.class) & @RunWith(SpringRunner.class)). So I just replaced

 RestTemplate restTemplate = restTemplateBuilder.build();

By

RestTemplate restTemplate = new RestTemplate();

And it wotk's !

Semantic Fuzzing with Zest
--------------------------
Test name:            com.example.demo.controller.PersonControllerFuzz#testCreatePerson
Results directory:    /full/path/to/api/javafuzzapi/target/fuzz-results/com.example.demo.controller.PersonControllerFuzz/testCreatePerson
Elapsed time:         1m 0s (max 1m 0s)
Number of executions: 5 835 (no trial limit)
Valid inputs:         5 835 (100,00%)
Cycles completed:     0
Unique failures:      0
Queue size:           45 (0 favored last cycle)
Current parent input: 8 (favored) {60/940 mutations}
Execution speed:      98/sec now | 97/sec overall
Total coverage:       4 616 branches (7,04% of map)
Valid coverage:       4 616 branches (7,04% of map)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:00 min
[INFO] Finished at: 2024-07-23T14:49:58+02:00
[INFO] ------------------------------------------------------------------------

Thank you so much for the help and explanations I hope this topic will help others !

[rohanpadhye]

Thanks for the update / follow up. Good luck!