This is the artifact of the research paper, "Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source", at ICSE 2022.
FreeFuzz is the first approach to fuzzing DL libraries via mining from open source. It collects code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to collect the dynamic information for each covered API. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API.
This is the FreeFuzz's implementation for testing PyTorch and TensorFlow.
ulimit -n 64000
to adjust the limit that the system resources a process may use. You can see this document for more details.requirements.txt
and run pip install -r requirements.txt
to install themRun the following command to load the database.
mongorestore dump/
src/instrumentation/{torch, tensorflow}
to see how to intrument the dynamic information and add them into the databasecd src && python preprocess/process_data.py torch
or for TensorFlow
cd src && python preprocess/process_data.py tf
There are some hyper-parameters in FreeFuzz and they could be easily configured as follows.
In src/config/demo.conf
:
[mongodb]
# your-mongodb-server
host = 127.0.0.1
# mongodb port
port = 27017
# name of pytorch database
torch_database = freefuzz-torch
# name of tensorflow database
tf_database = freefuzz-tf
[output]
# output directory for pytorch
torch_output = torch-output
# output directory for tensorflow
tf_output = tf-output
[oracle]
# enable crash oracle
enable_crash = true
# enable cuda oracle
enable_cuda = true
# enable precision oracle
enable_precision = true
# float difference bound: if |a-b| > bound, a is different than b
float_difference_bound = 1e-5
# max time bound: if time(low_precision) > bound * time(high_precision),
# it will be considered as a potential bug
max_time_bound = 10
# only consider the call with time(call) > time_thresold
time_thresold = 1e-3
[mutation]
enable_value_mutation = true
enable_type_mutation = true
enable_db_mutation = true
# the number of times each api is executed
each_api_run_times = 1000
After finishing above steps, run the following command to start FreeFuzz to test PyTorch
cd src && python FreeFuzz.py --conf demo_torch.conf
Or run this command to test TensorFlow
cd src && python FreeFuzz.py --conf demo_tf.conf
To run the full experiment, run the following command
cd src && python FreeFuzz.py --conf expr.conf
If you want to use another configuration file, you can put it in src/config
.
Note that you should specify the configuration file you want to use.
src/config/skip_torch.txt
or src/config/skip_tf
.