search

HexHive / magma

A ground-truth fuzzing benchmark suite based on real programs with real bugs.
https://hexhive.epfl.ch/magma
289 stars 84 forks source link

Blank workdir/crashes and null in survival_time/ci_triggered #160

Closed cloudymonstera closed 11 months ago

cloudymonstera commented 11 months ago

Hi there, every step of run.sh works as defined in captainrc, then unique crashes are presented in log/, monitor/ is derived and poc/ is filled, except that the directory named workdir/crashes is blank. Based on execution results, exp2json and survival_analysis are performed, while I just get initial survival_time/ci_reached and null survival_time/ci_triggered as follows:

target,program,bug,fuzzer,survival_time_reached,survival_ci_reached,survival_time_triggered,survival_ci_triggered
libpng,libpng_read_fuzzer,PNG001,afl,1.0,0.0,,
libpng,libpng_read_fuzzer,PNG001,aflfast,1.0,0.0,,
libpng,libpng_read_fuzzer,PNG001,fairfuzz,1.0,0.0,,
libpng,libpng_read_fuzzer,PNG001,moptafl,1.0,0.0,,
libpng,libpng_read_fuzzer,PNG003,afl,1.0,0.0,1.0,0.0
libpng,libpng_read_fuzzer,PNG003,aflfast,1.0,0.0,1.0,0.0
libpng,libpng_read_fuzzer,PNG003,fairfuzz,1.0,0.0,1.0,0.0
libpng,libpng_read_fuzzer,PNG003,moptafl,1.0,0.0,1.0,0.0

Any suggestions are welcome and appreciated. Thanks!

cloudymonstera commented 11 months ago

I find that crashes is compressed, the problem is partially solved. Once the trial_length is increased, the trends of survival_time/ci_reached are appeared. 

libxml2,xmllint,XML008,afl,99.39999999999999,4.611209819559038,,
libxml2,xmllint,XML008,aflfast,100.0,0.0,,
libxml2,xmllint,XML008,aflplusplus,99.89999999999999,1.1259342787202953,,
libxml2,xmllint,XML008,moptafl,99.8,1.1841385054111764,,
libxml2,libxml2_xml_read_memory_fuzzer,XML009,afl,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML009,aflfast,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML009,fairfuzz,99.14999999999995,9.570441365573751,,
libxml2,libxml2_xml_read_memory_fuzzer,XML009,moptafl,97.44999999999996,16.408197097792574,,
libxml2,xmllint,XML009,afl,97.44999999999996,16.408197097792574,,
libxml2,xmllint,XML009,aflfast,97.44999999999996,16.408197097792574,,
libxml2,xmllint,XML009,aflplusplus,99.09999999999995,10.133408508493574,,
libxml2,xmllint,XML009,fairfuzz,99.14999999999995,9.570441365573751,,
libxml2,xmllint,XML009,moptafl,97.44999999999996,16.408197097792574,100.0,0.0
libxml2,libxml2_xml_read_memory_fuzzer,XML011,afl,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML011,aflfast,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML011,fairfuzz,99.14999999999995,9.570441365573751,,
libxml2,libxml2_xml_read_memory_fuzzer,XML011,moptafl,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML012,afl,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML012,aflfast,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML012,fairfuzz,99.14999999999995,9.570441365573751,,
libxml2,libxml2_xml_read_memory_fuzzer,XML012,moptafl,97.44999999999996,16.408197097792574,,
libxml2,xmllint,XML012,afl,97.44999999999996,16.408197097792574,,
libxml2,xmllint,XML012,aflfast,97.44999999999996,16.408197097792574,,
libxml2,xmllint,XML012,aflplusplus,99.19999999999996,9.00747422977193,,
libxml2,xmllint,XML012,fairfuzz,99.14999999999995,9.570441365573751,,
libxml2,xmllint,XML012,moptafl,97.44999999999996,16.408197097792574,,
libxml2,libxml2_xml_read_memory_fuzzer,XML017,afl,97.29999999999995,17.37338516196561,97.59999999999997,15.443009033216587
libxml2,libxml2_xml_read_memory_fuzzer,XML017,aflfast,97.29999999999995,17.37338516196561,97.59999999999997,15.443009033216587
libxml2,libxml2_xml_read_memory_fuzzer,XML017,fairfuzz,99.09999999999995,10.133408508493574,99.19999999999996,9.00747422977193
libxml2,libxml2_xml_read_memory_fuzzer,XML017,moptafl,97.29999999999995,17.37338516196561,97.59999999999997,15.443009033216587
libxml2,xmllint,XML017,afl,97.29999999999995,17.37338516196561,99.2,5.48800000000021
libxml2,xmllint,XML017,aflfast,97.29999999999995,17.37338516196561,98.89999999999998,6.142810106132419
libxml2,xmllint,XML017,aflplusplus,99.14999999999995,9.570441365573751,99.59999999999998,4.503737114885577
libxml2,xmllint,XML017,fairfuzz,99.09999999999995,10.133408508493574,99.64999999999998,3.9407699755263526
libxml2,xmllint,XML017,moptafl,97.29999999999995,17.37338516196561,98.19999999999997,11.582256774912333

Nevertheless, survival_time/ci_triggered are still blank often, it seems like no bug is triggered since "monitor file contains no rows".