[memcached] Coverage build failure

[bxa-stripe]

Similar to a few other previous projects (#10717, #11081, #11010), memcached is failing to build coverage:

Step #5: [/corpus/fuzzer_proxy.zip]
Step #5:   End-of-central-directory signature not found.  Either this file is not
Step #5:   a zipfile, or it constitutes one disk of a multi-part archive.  In the
Step #5:   latter case the central directory and zipfile comment will be found on
Step #5:   the last disk(s) of this archive.
Step #5: unzip:  cannot find zipfile directory in one of /corpus/fuzzer_proxy.zip or
Step #5:         /corpus/fuzzer_proxy.zip.zip, and cannot find /corpus/fuzzer_proxy.zip.ZIP, period.
Step #5: Failed to unpack the corpus for fuzzer_proxy. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.

The fuzzer itself appears to be running, and I see a corpus in memcached-corpus.clusterfuzz-external.appspot.com. I also can't replicate this issue locally. Any ideas?

[oliverchang]

digging into this one, it looks like our corpus pruning jobs are timing out:

2023-11-12 21:39:48,353 - run_bot - ERROR - Merging new testcases timed out.
NoneType: None
2023-11-12 21:39:48,361 - run_bot - WARNING - Merge failed.
Traceback (most recent call last):
  File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/fuzzers/libFuzzer/engine.py", line 234, in _merge_new_units
    libfuzzer.DEFAULT_MERGE_TIMEOUT))
  File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/fuzzers/libFuzzer/engine.py", line 450, in _minimize_corpus_two_step
    reproducers_dir, max_time)
  File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/fuzzers/libFuzzer/engine.py", line 531, in minimize_corpus
    raise MergeError('Merging new testcases failed.')
clusterfuzz._internal.bot.fuzzers.libFuzzer.engine.MergeError: Merging new testcases failed.

Seeing what we can do to mitigate this

[bxa-stripe]

Hey @oliverchang, was there any movement on this?

[nathaniel-brough]

Looks like this is now an issue with surrealdb as well https://oss-fuzz-build-logs.storage.googleapis.com/index.html#surrealdb. Looks like there is a similar error;

Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: [/corpus/fuzz_structured_executor.zip]
Step #5:   End-of-central-directory signature not found.  Either this file is not
Step #5:   a zipfile, or it constitutes one disk of a multi-part archive.  In the
Step #5:   latter case the central directory and zipfile comment will be found on
Step #5:   the last disk(s) of this archive.
Step #5: unzip:  cannot find zipfile directory in one of /corpus/fuzz_structured_executor.zip or
Step #5:         /corpus/fuzz_structured_executor.zip.zip, and cannot find /corpus/fuzz_structured_executor.zip.ZIP, period.
Step #5: Failed to unpack the corpus for fuzz_structured_executor. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.
Step #5: ********************************************************************************
Step #5: Code coverage report generation failed.
Step #5: To reproduce, run:
Step #5: python infra/helper.py build_image surrealdb
Step #5: python infra/helper.py build_fuzzers --sanitizer coverage surrealdb
Step #5: python infra/helper.py coverage surrealdb
Step #5: ********************************************************************

[bxa-stripe]

👋 Are there any updates on this issue? Seems like several other projects are experiencing this.

[flavorjones]

Nokogiri is having this problem as well. It was fixed for a while after https://github.com/google/oss-fuzz/issues/11081#issuecomment-1763259458 but has been failing regularly since late April 2024. Example log