search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.54k stars 2.23k forks source link

CIFuzz issue only happens in CI, cannot reproduce #9470

Closed roidelapluie closed 1 year ago

roidelapluie commented 1 year ago

I see in some PRs fuzzing issues.

Example: https://github.com/prometheus/prometheus/actions/runs/3966052189/jobs/6796427069

/usr/bin/docker run --name ea425b91f0a59898e34670ab4e9a26c7903c86_01154b --label ea425b --workdir /github/workspace --rm -e "INPUT_OSS-FUZZ-PROJECT-NAME" -e "INPUT_FUZZ-SECONDS" -e "INPUT_DRY-RUN" -e "INPUT_LANGUAGE" -e "INPUT_SANITIZER" -e "INPUT_MODE" -e "INPUT_GITHUB-TOKEN" -e "INPUT_REPORT-UNREPRODUCIBLE-CRASHES" -e "INPUT_MINIMIZE-CRASHES" -e "INPUT_PARALLEL-FUZZING" -e "OSS_FUZZ_PROJECT_NAME" -e "LANGUAGE" -e "FUZZ_SECONDS" -e "DRY_RUN" -e "SANITIZER" -e "MODE" -e "GITHUB_TOKEN" -e "LOW_DISK_SPACE" -e "REPORT_UNREPRODUCIBLE_CRASHES" -e "MINIMIZE_CRASHES" -e "CIFUZZ_DEBUG" -e "CFL_PLATFORM" -e "PARALLEL_FUZZING" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/prometheus/prometheus":"/github/workspace" ea425b:91f0a59898e34670ab4e9a26c7903c86
2023-01-20 08:48:26,512 - root - DEBUG - Is github: True.
2023-01-20 08:48:26,515 - root - DEBUG - base_commit: None
2023-01-20 08:48:26,515 - root - DEBUG - pr_ref: refs/pull/11871/merge
2023-01-20 08:48:26,515 - root - DEBUG - No PROJECT_SRC_PATH.
2023-01-20 08:48:26,516 - root - INFO - Deleting builder docker images to save disk space.
2023-01-20 08:48:27,638 - root - DEBUG - Stderr of command "docker rmi -f gcr.io/oss-fuzz/prometheus gcr.io/oss-fuzz-base/base-builder gcr.io/oss-fuzz-base/base-builder-go gcr.io/oss-fuzz-base/base-builder-javascript gcr.io/oss-fuzz-base/base-builder-jvm gcr.io/oss-fuzz-base/base-builder-python gcr.io/oss-fuzz-base/base-builder-rust gcr.io/oss-fuzz-base/base-builder-swift" is: Error: No such image: gcr.io/oss-fuzz-base/base-builder
Error: No such image: gcr.io/oss-fuzz-base/base-builder-javascript
Error: No such image: gcr.io/oss-fuzz-base/base-builder-jvm
Error: No such image: gcr.io/oss-fuzz-base/base-builder-python
Error: No such image: gcr.io/oss-fuzz-base/base-builder-rust
Error: No such image: gcr.io/oss-fuzz-base/base-builder-swift
.
2023-01-20 08:48:27,694 - root - INFO - ClusterFuzzDeployment: <clusterfuzz_deployment.OSSFuzz object at 0x7f5d04045e20>.
2023-01-20 08:48:27,694 - root - INFO - run fuzzers MODE is: code-change. Runner: <run_fuzzers.CiFuzzTargetRunner object at 0x7f5d04045dc0>.
2023-01-20 08:48:27,695 - root - INFO - Using address sanitizer.
2023-01-20 08:48:27,789 - root - INFO - Fuzz targets: ['/github/workspace/build-out/fuzzParseExpr', '/github/workspace/build-out/fuzzParseOpenMetric', '/github/workspace/build-out/fuzzParseMetricSelector', '/github/workspace/build-out/fuzzParseMetric']
2023-01-20 08:48:27,790 - root - INFO - Running fuzzer: fuzzParseExpr.
2023-01-20 08:48:27,790 - root - INFO - Downloading corpus from OSS-Fuzz: https://storage.googleapis.com/prometheus-backup.clusterfuzz-external.appspot.com/corpus/libFuzzer/prometheus_fuzzParseExpr/public.zip
2023-01-20 08:48:27,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): storage.googleapis.com:443
2023-01-20 08:48:27,819 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "GET /prometheus-backup.clusterfuzz-external.appspot.com/corpus/libFuzzer/prometheus_fuzzParseExpr/public.zip HTTP/1.1" 200 2263724
2023-01-20 08:48:28,670 - root - INFO - Starting fuzzing
/github/workspace/build-out/fuzzParseExpr -timeout=25 -rss_limit_mb=2560 -len_control=0 -seed=1337 -artifact_prefix=/tmp/tmpkw58_qph/ -max_total_time=150 -print_final_stats=1 /github/workspace/cifuzz-corpus/fuzzParseExpr >fuzz-0.log 2>&1
2023-01-20 08:51:09,186 - root - INFO - Fuzzer: fuzzParseExpr. Detected bug.
2023-01-20 08:51:09,187 - root - INFO - Trying to reproduce crash using: /tmp/tmpkw58_qph/tmpk_oz1chp.
2023-01-20 08:51:10,332 - root - INFO - Reproduce command returned: 1. Reproducible on /github/workspace/build-out/fuzzParseExpr.
2023-01-20 08:51:10,332 - root - INFO - Crash is reproducible.
2023-01-20 08:51:10,412 - root - INFO - Downloading latest build.
2023-01-20 08:51:10,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): storage.googleapis.com:443
2023-01-20 08:51:10,481 - urllib3.connectionpool - DEBUG - https://storage.googleapis.com:443 "GET /clusterfuzz-builds/prometheus/prometheus-address-202301200610.zip HTTP/1.1" 200 53189256
2023-01-20 08:51:11,826 - root - INFO - Done downloading latest build.
2023-01-20 08:51:11,827 - root - INFO - Trying to reproduce crash using: /tmp/tmpkw58_qph/tmpk_oz1chp.
2023-01-20 08:51:14,118 - root - INFO - Reproduce command returned: 0. Not reproducible on /github/workspace/cifuzz-prev-build/fuzzParseExpr.
2023-01-20 08:51:14,119 - root - INFO - The crash is not reproducible on previous build. Code change (pr/commit) introduced crash.
2023-01-20 08:51:14,120 - root - INFO - SAVING CRASH
2023-01-20 08:51:14,121 - root - INFO - NOT MINIMIZED
2023-01-20 08:51:14,121 - root - INFO - Deleting corpus and seed corpus of fuzzParseExpr to save disk.
2023-01-20 08:51:14,316 - root - INFO - Deleting fuzz target: fuzzParseExpr.
2023-01-20 08:51:14,317 - root - INFO - Done deleting.
2023-01-20 08:51:14,318 - root - INFO - Bug found. Stopping fuzzing.
2023-01-20 08:51:14,318 - root - INFO - Not uploading crashes because on OSS-Fuzz.
2023-01-20 08:51:14,319 - root - INFO - Bug found.

However, the code fuzzed is the same as the main branch. And when I run the test case locally it passes:

$ python3 infra/helper.py reproduce prometheus fuzzParseExpr ./fuzzParseExpr/address/tmpk_oz1chp
INFO:root:Running: docker run --rm --privileged --shm-size=2g --platform linux/amd64 -i -e HELPER=True -e ARCHITECTURE=x86_64 -v /home/roidelapluie/dev/oss-fuzz/build/out/prometheus:/out -v /home/roidelapluie/dev/oss-fuzz/fuzzParseExpr/address/tmpk_oz1chp:/testcase -t gcr.io/oss-fuzz-base/base-runner reproduce fuzzParseExpr -runs=100.
+ FUZZER=fuzzParseExpr
+ shift
+ '[' '!' -v TESTCASE ']'
+ TESTCASE=/testcase
+ '[' '!' -f /testcase ']'
+ export RUN_FUZZER_MODE=interactive
+ RUN_FUZZER_MODE=interactive
+ export FUZZING_ENGINE=libfuzzer
+ FUZZING_ENGINE=libfuzzer
+ export SKIP_SEED_CORPUS=1
+ SKIP_SEED_CORPUS=1
+ run_fuzzer fuzzParseExpr -runs=100 /testcase
/out/fuzzParseExpr -rss_limit_mb=2560 -timeout=25 -runs=100 /testcase < /dev/null
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2967943699
INFO: Loaded 1 modules   (80022 inline 8-bit counters): 80022 [0x237d008, 0x239089e), 
INFO: Loaded 1 PC tables (80022 PCs): 80022 [0x10c000080000,0x10c0001b8960), 
/out/fuzzParseExpr: Running 1 inputs 100 time(s) each.
Running: /testcase
Executed /testcase in 24 ms
***
*** NOTE: fuzzing was not performed, you have only
***       executed the target code on a fixed set of inputs.
***

Any cloud would be welcome. I have seen similar issues here but I don't think this is related (#9318)

jonathanmetzman commented 1 year ago

Sorry for this issue. I just got back from vacation so I think I will deploy the fix for https://github.com/google/oss-fuzz/issues/9318 on monday. My hunch is these problems are related but if they aren't ill find out then and fix it.

jonathanmetzman commented 1 year ago

Is this still happening to you?

maflcko commented 1 year ago

Can be closed?

jonathanmetzman commented 1 year ago

Yeah. closing due to inactivity