Garbled `target_lists` break coverage builds for multiple projects

[fmeum]

(https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=36429)

Without any changes to its build setup and no recent changes to the upstream repo, the coverage build for jackson-dataformat-xml is now failing in step 5 (full log):

Starting Step #5
Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: [/corpus/塭汆畺穥爊.zip]
Step #5:   End-of-central-directory signature not found.  Either this file is not
Step #5:   a zipfile, or it constitutes one disk of a multi-part archive.  In the
Step #5:   latter case the central directory and zipfile comment will be found on
Step #5:   the last disk(s) of this archive.
Step #5: unzip:  cannot find zipfile directory in one of /corpus/塭汆畺穥爊.zip or
Step #5:         /corpus/塭汆畺穥爊.zip.zip, and cannot find /corpus/塭汆畺穥爊.zip.ZIP, period.
Step #5: Failed to unpack the corpus for 塭汆畺穥爊. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.

The coverage build works flawlessly locally and the target list at https://storage.googleapis.com/clusterfuzz-builds/jackson-dataformat-xml/targets.list.address also looks correct.

I tried to decode that garbled filename with various encodings, but still have no clue which part of the stack is responsible for this issue.

[jonathanmetzman]

Weird. This doesn't happen with the helper.py script.

[fmeum]

@jonathanmetzman The issue is more widespread and also affects other projects, but non-deterministically. Coverage builds for these projects also failed in the last week due to this issue: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#apache-commons https://oss-fuzz-build-logs.storage.googleapis.com/index.html#javaparser

Is it possible that the GET request for the target list in the GCloud bucket has an unexpected response, perhaps due to unspecified character encoding, redirects or rate limiting? It could be helpful to log the full HTTP response here.

[jonathanmetzman]

Hmm...I can't seem to trigger this by manually scheduling a coverage job. Maybe it is a rate limiting issue.

[fmeum]

This issue also affects non-Java projects (e.g. uwebsockets) and started to appear around July 21.

[jonathanmetzman]

@asraa since you're current sheriff could you please look into this? 7 days ago was when we started fixing the issue with preserving state in HOME: https://github.com/google/oss-fuzz/pull/6079 and https://github.com/google/oss-fuzz/pull/6069 But I don't see how this is related.

[athre0z]

We're having the same issue with the Zydis coverage build: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=36433

[fmeum]

@asraa Did you have a chance to look into this?

[asraa]

Taking a look now

[asraa]

Is it possible that the GET request for the target list in the GCloud bucket has an unexpected response, perhaps due to unspecified character encoding, redirects or rate limiting? It could be helpful to log the full HTTP response here.

I tried locally to slam the gcloud bucket with the same method as used in the build_lib, and don't think this is the issue. (Regardless it might not hurt to explicitly add an accept encoding). Still tracing this down