Support custom local docker registry for local experiments.

[Anothonio]

Hi

I started to run a local experiment as described here and here .

I run the following command :

PYTHONPATH=. python3 experiment/run_experiment.py --experiment-config experiment-config.yaml --benchmarks freetype2-2017 bloaty_fuzz_target --experiment-name $EXPERIMENT_NAME --fuzzers afl libfuzzer the experiment-config.yaml has the following content :

# The number of trials of a fuzzer-benchmark pair.
trials: 5

# The amount of time in seconds that each trial is run for.
# 1 day = 24 * 60 * 60 = 86400
max_total_time: 1800

# The location of your docker registry.
docker_registry: lab-server:5000

# The local experiment folder that will store most of the experiment data.
# Please use an absolute path.
experiment_filestore: /tmp/experiment-data

# The local report folder where HTML reports and summary data will be stored.
# Please use an absolute path.
report_filestore: /tmp/report-data

But I got the following error :

ERROR:root:Config does not contain "cloud_project".
ERROR:root:Config does not contain "cloud_compute_zone".
ERROR:root:Config parameter "experiment_filestore" is "/tmp/experiment-data". It must start with gs:// when running on Google Cloud.
ERROR:root:Config parameter "report_filestore" is "/tmp/report-data". It must start with gs:// when running on Google Cloud.
Traceback (most recent call last):
  File "experiment/run_experiment.py", line 483, in <module>
    sys.exit(main())
  File "experiment/run_experiment.py", line 478, in main
    no_dictionaries=args.no_dictionaries)
  File "experiment/run_experiment.py", line 214, in start_experiment
    config = read_and_validate_experiment_config(config_filename)
  File "experiment/run_experiment.py", line 132, in read_and_validate_experiment_config
    raise ValidationError('Config: %s is invalid.' % config_filename)
__main__.ValidationError: Config: experiment-config.yaml is invalid.

I did everything as documented but I came up with this .

I did not understand the following in config file : docker_registry: lab-server:5000

Should I setup a docker_registry ? if yes how to do this ?

thnx

[inferno-chromium]

experiment: test-experiment
trials: 2
max_total_time: 900
docker_registry: gcr.io/fuzzbench
experiment_filestore: /tmp/fuzzbench-data
report_filestore: /tmp/fuzzbench-reports
local_experiment: true

ah docs are missing "local_experiment: true", let me add it. also docker_registry, try "gcr.io/fuzzbench"

[inferno-chromium]

Doc flag added in https://github.com/google/fuzzbench/commit/1cc0c99f08a1dc7259105be587eea0b292a284df

[inferno-chromium]

For docker registry, use gcr.io/fuzzbench. I will work with @lszekeres to see why lab-server:5000 was recommended, and how to set this up in gcr.io/fuzzbench context.

[inferno-chromium]

Please just use "gcr.io/fuzzbench" in that field, we will add the support in some time.

[Anothonio]

Thanks for the response, it worked!

Well, after running the test I have the following error for generating reports :

INFO:root:Generating coverage report for benchmark: libpng-1.2.56 fuzzer: afl.
INFO:root:Merging profdata for fuzzer: afl,benchmark: libpng-1.2.56.
ERROR:root:Profdata files merging failed.
ERROR:root:Error occurred when generating coverage report. Extras: {'traceback': 'Traceback (most recent call last):\n  File "/work/src/experiment/coverage_utils.py", line 73, in generate_coverage_report\n    coverage_reporter.generate_coverage_summary_json()\n  File "/work/src/experiment/coverage_utils.py", line 143, in generate_coverage_summary_json\n    summary_only=False)\n  File "/work/src/experiment/coverage_utils.py", line 265, in generate_json_summary\n    with open(output_file, \'w\') as dst_file:\nFileNotFoundError: [Errno 2] No such file or directory: \'/work/measurement-folders/libpng-1.2.56-afl/merged.json\'\n'}
INFO:root:Generating coverage report for benchmark: libpng-1.2.56 fuzzer: honggfuzz.
INFO:root:Merging profdata for fuzzer: honggfuzz,benchmark: libpng-1.2.56.
ERROR:root:Profdata files merging failed.
ERROR:root:Error occurred when generating coverage report. Extras: {'traceback': 'Traceback (most recent call last):\n  File "/work/src/experiment/coverage_utils.py", line 73, in generate_coverage_report\n    coverage_reporter.generate_coverage_summary_json()\n  File "/work/src/experiment/coverage_utils.py", line 143, in generate_coverage_summary_json\n    summary_only=False)\n  File "/work/src/experiment/coverage_utils.py", line 265, in generate_json_summary\n    with open(output_file, \'w\') as dst_file:\nFileNotFoundError: [Errno 2] No such file or directory: \'/work/measurement-folders/libpng-1.2.56-honggfuzz/merged.json\'\n'}
INFO:root:Generating coverage report for benchmark: libpng-1.2.56 fuzzer: libfuzzer.
INFO:root:Merging profdata for fuzzer: libfuzzer,benchmark: libpng-1.2.56.
ERROR:root:Profdata files merging failed.
ERROR:root:Error occurred when generating coverage report. Extras: {'traceback': 'Traceback (most recent call last):\n  File "/work/src/experiment/coverage_utils.py", line 73, in generate_coverage_report\n    coverage_reporter.generate_coverage_summary_json()\n  File "/work/src/experiment/coverage_utils.py", line 143, in generate_coverage_summary_json\n    summary_only=False)\n  File "/work/src/experiment/coverage_utils.py", line 265, in generate_json_summary\n    with open(output_file, \'w\') as dst_file:\nFileNotFoundError: [Errno 2] No such file or directory: \'/work/measurement-folders/libpng-1.2.56-libfuzzer/merged.json\'\n'}
INFO:root:Finished generating coverage reports.
INFO:root:Finished measuring.

Any solution to this ?

[BharathMonash]

@inferno-chromium, same issue while running experiment locally now.

The merging of profile data fails since it is unable to locate the destination path specified to place the merged.profdata in. Coverage reporting entirely fails since profile data is missing.

[Anothonio]

I am not sure but I think when the max_total_timeis lower than 500, then this happend .

@BharathMonash Please test with 900as the max_total_timeso it probably will be run and generate reports. (I think!)

Please confirm if this is the case for you but I am still waiting for a response from google fuzzbench guys about this .

[BharathMonash]

I am not sure but I think when the max_total_timeis lower than 500, then this happend . @BharathMonash Please test with 900as the max_total_timeso it probably will be run and generate reports. (I think!) Please confirm if this is the case for you but I am still waiting for a response from google fuzzbench guys about this .

@Anothonio, I ran with max_total_time: 900, still getting the same error while generating reports. Measuring runs properly.

coverage_reporter.generate_coverage_summary_json()\n  File "/work/src/experiment/coverage_utils.py", line 143, 
in generate_coverage_summary_json\n summary_only=False)\n  File "/work/src/experiment/coverage_utils.py", 
line 265, ingenerate_json_summary\n with open(output_file, \'w\') as dst_file:\nFileNotFoundError: 
[Errno 2] No such file or directory: \'/work/measurement-folders/libpng-1.2.56-afl/merged.json\'\n'}```

dst_file:\nFileNotFoundError: [Errno 2] No such file or directory: 
\'/work/measurement-folders/libpng-1.2.56-afl/merged.json\'\n'}

[mboehme]

The corpus is copied at exactly 15 minutes (900sec). Maybe you can try to set max_total_time just a bit higher than 900, say 1000?

[inferno-chromium]

First question, is your experiment report generated at /tmp/fuzzbench-reports//index.html and can you try to run python3 -m http.server on that dir to view report. That error just means you probably have an empty corpus, can you try running with max_total_time: 3600 and if that does not work, try other benchmark. you need more time for a corpus to be generated. non-empty corpus -> profraw -> combined profdata -> summary json

[BharathMonash]

@inferno-chromium and @mboehme, yes the reports and plots (svg) are generated at: /tmp/fuzzbench-reports/<experiment_name>/index.html and I'm able to view the html report. I'l try a higher time as suggested and see if the issue persists.