search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 270 forks source link

Verify recent error cause #2011

Closed DonggeLiu closed 3 months ago

DonggeLiu commented 3 months ago

Recent PR experiments experience similar errors. This PR intends to verify if the error is caused by our recent code changes.

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-03-dg --fuzzers libfuzzer --benchmarks libxml2_xml

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-03-test --fuzzers libfuzzer --benchmarks libxml2_xml

DonggeLiu commented 3 months ago

Experiment 2024-08-03-test data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

vanhauser-thc commented 3 months ago

weird your experiment is not coming up at https://www.fuzzbench.com/reports/experimental/index.html but is available here - https://storage.googleapis.com/www.fuzzbench.com/reports/2024-08-03-test/index.html

DanBlackwell commented 3 months ago

@DonggeLiu I think there may be a correlation with merging results. One of my experiments (here) on 1 aug ran fine merging disabled, but the ones with merging enabled failed to run the coverage checker... I see your run up above ran fine with merging disabled too.

DonggeLiu commented 3 months ago

weird your experiment is not coming up at https://www.fuzzbench.com/reports/experimental/index.html but is available here - https://storage.googleapis.com/www.fuzzbench.com/reports/2024-08-03-test/index.html

Yep, I recall @jonathanmetzman has changed the logic.

DonggeLiu commented 3 months ago

@DonggeLiu I think there may be a correlation with merging results. One of my experiments (here) on 1 aug ran fine merging disabled, but the ones with merging enabled failed to run the coverage checker... I see your run up above ran fine with merging disabled too.

Interesting observation, thanks! Let me run another one with merge enabled.

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-05-test --fuzzers libfuzzer_test --benchmarks libxml2_xml

DonggeLiu commented 3 months ago

Experiment 2024-08-05-test data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

DanBlackwell commented 3 months ago

@DonggeLiu I think there may be a correlation with merging results. One of my experiments (here) on 1 aug ran fine merging disabled, but the ones with merging enabled failed to run the coverage checker... I see your run up above ran fine with merging disabled too.

Interesting observation, thanks! Let me run another one with merge enabled.

Ok, bad news is it doesn't seem to be quite so simple. This failed to run with merging disabled :/

DonggeLiu commented 3 months ago

Ok, bad news is it doesn't seem to be quite so simple. This failed to run with merging disabled :/

Yep, the experiment did not generate a report at all. Merging only happens at the very end stage after the experiment is completed.

DonggeLiu commented 3 months ago

Hi @DanBlackwell, The two experiments in this PR seem to be working as expected, so is #1966. The cloud log shows the report generation failure in 2024-08-05-dgfuzz is due to Profdata files Prmerging failed. from https://github.com/google/fuzzbench/blob/d8a77233f3919477648c98d2c8228b530b7036ba/experiment/measurer/coverage_utils.py#L134-L136

There are also many Error watching metadata: context canceled from gcloud rather than FB or your fuzzer.

I will cancel all experiment instances for 2024-08-05-dgfuzz for now. Meanwhile, would you mind rebasing your PR on this one, just in case the error is related to the commit that this PR reverted? Thanks!

DanBlackwell commented 3 months ago

Hi @DanBlackwell, The two experiments in this PR seem to be working as expected, so is #1966. The cloud log shows the report generation failure in 2024-08-05-dgfuzz is due to Profdata files Prmerging failed. from

https://github.com/google/fuzzbench/blob/d8a77233f3919477648c98d2c8228b530b7036ba/experiment/measurer/coverage_utils.py#L134-L136

There are also many Error watching metadata: context canceled from gcloud rather than FB or your fuzzer.

I will cancel all experiment instances for 2024-08-05-dgfuzz for now. Meanwhile, would you mind rebasing your PR on this one, just in case the error is related to the commit that this PR reverted? Thanks!

Hi @DonggeLiu , I believe it's only possible to rebase onto branches in the parent repo; not PRs. As an alternative I've created a squashed patch of #2000, and you should be able to just git patch apply it in this PR, then commit and push it up if you want to test that the issue is resolved.

Here's the patch file: dgfuzz_commit.patch

DonggeLiu commented 3 months ago

Hi @DonggeLiu , I believe it's only possible to rebase onto branches in the parent repo; not PRs.

I reckon clicking the Edit button (on the top, to the right of the PR title) works?

image image

Let me know if this is not available to you though, I can do it for you (if you don't mind).

With that said, seeing libfuzzer works on your PR and aflpp work in https://github.com/google/fuzzbench/pull/1966, I wonder if other reasons caused the failure.

DanBlackwell commented 3 months ago

Hi @DonggeLiu , I believe it's only possible to rebase onto branches in the parent repo; not PRs.

I reckon clicking the Edit button (on the top, to the right of the PR title) works? image

image

Let me know if this is not available to you though, I can do it for you (if you don't mind).

My bad, you are correct; I have rebased it now.

With that said, seeing libfuzzer works on your PR and aflpp work in #1966, I wonder if other reasons caused the failure.

Hmm, is there anything in the logs for the measurer at all?

DonggeLiu commented 3 months ago

Hmm, is there anything in the logs for the measurer at all?

The report shows the coverage of the fuzzer, so I presume the measurer must be running? Please correct me if I missed anything : )

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-07-libfuzzer-zlib --fuzzers libfuzzer --benchmarks zlib_zlib_uncompress_fuzzer

DonggeLiu commented 3 months ago

I reckon we can conclude the error is not caused by FuzzBench infra? @DanBlackwell please feel free to reopen this if proven otherwise. Thanks : )