jonathanmetzman
commented
1 year ago Sorry for the late response, I don't have a great guess. Do you want to email me [my last name]@chromium.org the data.csv.gz so I can try to reproduce?
Open krz-max opened 1 year ago
jonathanmetzman
commented
1 year ago Sorry for the late response, I don't have a great guess. Do you want to email me [my last name]@chromium.org the data.csv.gz so I can try to reproduce?
krz-max
commented
1 year ago Thanks a lot for the reply, but due to the error, report data isn't generated automatically after the experiment ends.
So, I modify the funcion, add_bugs_covered_column(experiment_df) in /fuzzbench/analysis/data_utils.py around line 153
def add_bugs_covered_column(experiment_df):
"""Return a modified experiment df in which adds a |bugs_covered| column,
a cumulative count of bugs covered over time."""
# Immediately return to avoid the code producing bugs
experiment_df['bugs_covered'] = 0
return experiment_df
"""
if 'crash_key' not in experiment_df:
experiment_df['bugs_covered'] = 0
return experiment_df
"""
grouping2 = ['fuzzer', 'benchmark', 'trial_id']
grouping3 = ['fuzzer', 'benchmark', 'trial_id', 'time']
df = experiment_df.sort_values(grouping3)
# Bug Here
df['firsts'] = (
df.groupby(grouping2, group_keys=False).apply(is_unique_crash) &
~df.crash_key.isna())
df['bugs_cumsum'] = df.groupby(grouping2)['firsts'].transform('cumsum')
df['bugs_covered'] = (
df.groupby(grouping3)['bugs_cumsum'].transform('max').astype(int))
new_df = df.drop(columns=['bugs_cumsum', 'firsts'])
return new_dfI will reproduce my experiment using the modified function and send it to you later. If you want to run it yourself, my execution command is something like:
PYTHONPATH=. python3 experiment/run_experiment.py -b freetype2_ftfuzzer -c experiment-config.yaml -e <experiment name> -f aflplusplusAnd my experiment-config.yaml:
# The number of trials of a fuzzer-benchmark pair.
trials: 1
# The amount of time in seconds that each trial is run for.
# 1 day = 24 * 60 * 60 = 86400
max_total_time: 7200
# The location of the docker registry.
# FIXME: Support custom docker registry.
# See https://github.com/google/fuzzbench/issues/777
docker_registry: gcr.io/fuzzbench
# The local experiment folder that will store most of the experiment data.
# Please use an absolute path.
experiment_filestore: /tmp/experiment-data
# The local report folder where HTML reports and summary data will be stored.
# Please use an absolute path.
report_filestore: /tmp/report-data
# Flag that indicates this is a local experiment.
local_experiment: trueAdditional Information:
experiment_df, they are None. I am not sure if these would help.
FatPigeorz
commented
1 year ago same issue, have you fixed it?
steven-hh-ding
commented
3 months ago Ran into the same problem. If you ran two fuzzer on a benchmark, it magically went away though..
steven-hh-ding
commented
3 months ago
df['firsts'] = (
df.groupby(grouping2, group_keys=False).apply(is_unique_crash) &
~df.crash_key.isna())The problem is that group_keys=False is not working here when there is only one entry, df.groupby(grouping2, group_keys=False).apply(is_unique_crash) outputs:
Name: firsts, dtype: bool
c1 firsts 0
fuzzer benchmark trial_id
aflplusplus libxml2_xml 58 TrueIt includes group key rather than the expected output to work
0 True
Name: firsts, dtype: boolTherefore, the included group key does not fit the original index of df, raising incompatible index of inserted column with frame index
My temporary fix:
df = experiment_df.sort_values(grouping3)
c1 = df.groupby(grouping2, group_keys=False).apply(is_unique_crash)
c2 = ~df.crash_key.isna()
c1.index = c2.index
df['firsts'] = c1 & c2
I was running local experiment using fuzzers
aflplusplusand benchmarkscurl_curl_fuzzer_httpandbloaty_fuzz_targetI pass themake presubmitafter installingqtbase-dev5mentioned in this issue #1867My experiment was successful before(about 2 weeks ago) but after I reinstall and execute again, it keeps reporting as follow: (I think the WARNING is fine because there are same warnings even when my experiment is successful)
I'm not sure why it directly print out
\n, so I tried to format the error message below for better reference:I've removed ubuntu and set up the environment several times but it does not work. My environment:
I download the docker from here Appreciate for the help!