Closed vy closed 1 month ago
DavidKorczynski
commented
1 month ago Can you confirm that you can login to oss-fuzz.com with the email https://github.com/google/oss-fuzz/blob/c523269e8ce17c8039534c266b1c9b14169d5bb1/projects/log4j2/project.yaml#L15 ? it's not showing up on the dashboard once you're logged in?
vy
commented
1 month ago @DavidKorczynski, I can login (also my other Log4j colleague whom I added using auto_ccs), but the fuzzers showing up (libFuzzer_log4j2_Log4jFuzzer and libFuzzer_log4j2_Log4jSlf4jFuzzer) are still from the old implementation. Likewise, the associated bucket in the Cloud Storage still contains data from the old fuzzers.
Instead, as hinted in the following build log snippet, we are supposed to see totally different fuzzers, e.g., log4j-core-fuzz-test-PatternLayoutFuzzer:
...
Finished Step #5
Starting Step #6
Step #6: Already have image: gcr.io/oss-fuzz/log4j2
Step #6: adding: jazzer_agent_deploy.jar (deflated 10%)
Step #6: adding: jazzer_driver (deflated 69%)
Step #6: adding: jazzer_driver_with_sanitizer (deflated 9%)
Step #6: adding: json.dict (deflated 46%)
Step #6: adding: json_seed_corpus.zip (stored 0%)
Step #6: adding: llvm-symbolizer (deflated 66%)
Step #6: adding: log4j-core-fuzz-test-2.24.0-SNAPSHOT.jar (deflated 18%)
Step #6: adding: log4j-core-fuzz-test-PatternLayoutFuzzer (deflated 58%)
Step #6: adding: log4j-fuzz-test-2.24.0-SNAPSHOT.jar (deflated 19%)
Step #6: adding: log4j-layout-template-json-fuzz-test-2.24.0-SNAPSHOT.jar (deflated 21%)
Step #6: adding: log4j-layout-template-json-fuzz-test-JsonTemplateLayoutCodecFuzzer (deflated 59%)
Step #6: adding: log4j-layout-template-json-fuzz-test-JsonTemplateLayoutCodecFuzzer.dict (deflated 46%)
Step #6: adding: log4j-layout-template-json-fuzz-test-JsonTemplateLayoutCodecFuzzer_seed_corpus.zip (stored 0%)
Step #6: adding: log4j-layout-template-json-fuzz-test-JsonTemplateLayoutFuzzer (deflated 60%)
Step #6: adding: log4j-slf4j2-impl-fuzz-test-2.24.0-SNAPSHOT.jar (deflated 18%)
Step #6: adding: log4j-slf4j2-impl-fuzz-test-Slf4jToLog4jBridgeWithJsonTemplateLayoutFuzzer (deflated 59%)
Step #6: adding: log4j-slf4j2-impl-fuzz-test-Slf4jToLog4jBridgeWithPatternLayoutFuzzer (deflated 59%)
...I personally wanted to rename projects/log4j2 to projects/apache-logging-log4j2 in #12304, but asked not to do so. Would it be an idea to submit a new PR copying projects/log4j2 to projects/apache-logging-log4j2 and re-introducing it as a new project [with the hope that this will give the ClusterFuzz infrastructure a good kick to get things going]?
vy
commented
1 month ago I've just being pinged that coverage builds are failing!
...
Finished Step #4
Starting Step #5
Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: [/corpus/log4j-core-fuzz-test-PatternLayoutFuzzer.zip]
Step #5: End-of-central-directory signature not found. Either this file is not
Step #5: a zipfile, or it constitutes one disk of a multi-part archive. In the
Step #5: latter case the central directory and zipfile comment will be found on
Step #5: the last disk(s) of this archive.
Step #5: unzip: cannot find zipfile directory in one of /corpus/log4j-core-fuzz-test-PatternLayoutFuzzer.zip or
Step #5: /corpus/log4j-core-fuzz-test-PatternLayoutFuzzer.zip.zip, and cannot find /corpus/log4j-core-fuzz-test-PatternLayoutFuzzer.zip.ZIP, period.
Step #5: Failed to unpack the corpus for log4j-core-fuzz-test-PatternLayoutFuzzer. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.
...
Step #5: ********************************************************************************
Step #5: Code coverage report generation failed.
Step #5: To reproduce, run:
Step #5: python infra/helper.py build_image log4j2
Step #5: python infra/helper.py build_fuzzers --sanitizer coverage log4j2
Step #5: python infra/helper.py coverage log4j2
Step #5: ********************************************************************************
Finished Step #5
ERROR
ERROR: build step 5 "gcr.io/oss-fuzz-base/base-runner" failed: step exited with non-zero status: 1I was able to locally reproduce the absence of the corpus:
$ python infra/helper.py coverage log4j2
INFO:__main__:Downloading corpora for log4j2 project to /home/vy/Projects/google-oss-fuzz~master/build/corpus/log4j2.
CommandException: One or more URLs matched no objects.
CommandException: One or more URLs matched no objects.
CommandException: One or more URLs matched no objects.
CommandException: One or more URLs matched no objects.
CommandException: One or more URLs matched no objects.
WARNING:__main__:Corpus for log4j2_log4j-slf4j2-impl-fuzz-test-Slf4jToLog4jBridgeWithJsonTemplateLayoutFuzzer not found:
...Since contributed fuzzers are new, it makes sense that the corpus is empty. I don't understand why they are treated as errors and I don't know how to fix them either. @DavidKorczynski, any ideas?
vy
commented
1 month ago Trying to clone the corpus doesn't work due to insufficient rights:
$ gsutil cp -r \
gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_Log4jFuzzer \
gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j-core-fuzz-test-PatternLayoutFuzzer
Copying gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_Log4jFuzzer/000aeaa9dfccf425541f6b7cbcc4d10977e6052b [Content-Type=application/octet-stream]...
AccessDeniedException: 403 vol***@***.ci does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).Neither the "Upload testcase" screen is allowing me to choose one of the new targets, e.g., log4j-core-fuzz-test-PatternLayoutFuzzer:
DavidKorczynski
commented
1 month ago I don't understand why they are treated as errors and I don't know how to fix them either.
Yeah this is kind of a bug. I think what has happened is the fuzzers didn't run to generate a corpus before the code coverage build was set. Ultimately I think this should stabilize in the near future.
Otherwise, @jonathanmetzman may have more clues
DavidKorczynski
commented
1 month ago I personally wanted to rename
projects/log4j2toprojects/apache-logging-log4j2in #12304, but asked not to do so. Would it be an idea to submit a new PR copyingprojects/log4j2toprojects/apache-logging-log4j2and re-introducing it as a new project [with the hope that this will give the ClusterFuzz infrastructure a good kick to get things going]?
For now I prefer to keep things as is. Your current approach is similar to how other projects fix/add fuzzers -- it should be working without issue
jonathanmetzman
commented
1 month ago I think we're not properly handling deleted fuzzers.
jonathanmetzman
commented
1 month ago Trying to clone the corpus doesn't work due to insufficient rights:
$ gsutil cp -r \ gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_Log4jFuzzer \ gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j-core-fuzz-test-PatternLayoutFuzzer Copying gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_Log4jFuzzer/000aeaa9dfccf425541f6b7cbcc4d10977e6052b [Content-Type=application/octet-stream]... AccessDeniedException: 403 vol***@***.ci does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).Neither the "Upload testcase" screen is allowing me to choose one of the new targets, e.g.,
log4j-core-fuzz-test-PatternLayoutFuzzer:
This issue is fixed now. I see some of the fuzzers running: gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_log4j-layout-template-json-fuzz-test-JsonTemplateLayoutCodecFuzzer/
others seem to be exceptioning. E.g. log4j-core-fuzz-test-PatternLayoutFuzzer
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
INFO: Loaded 261 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.sanitizers.ClojureLangHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection
INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection
INFO: Loaded 46 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup
INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection
INFO: Loaded 48 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall
INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection
INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks
INFO: Loaded 12 hooks from com.code_intelligence.jazzer.sanitizers.ScriptEngineInjection
INFO: Loaded 3 hooks from com.code_intelligence.jazzer.sanitizers.ServerSideRequestForgery
INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection
INFO: Loaded 6 hooks from com.code_intelligence.jazzer.sanitizers.XPathInjection
INFO: Instrumented org.apache.logging.log4j.core.fuzz.PatternLayoutFuzzer (took 185 ms, size +3%)
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3325)
at java.base/java.lang.Class.getMethodsRecursive(Class.java:3466)
at java.base/java.lang.Class.getMethod0(Class.java:3452)
at java.base/java.lang.Class.getMethod(Class.java:2199)
at com.code_intelligence.jazzer.driver.ReflectionUtils.targetPublicStaticMethod(ReflectionUtils.java:27)
at com.code_intelligence.jazzer.driver.FuzzTargetFinder.findFuzzTargetByMethodName(FuzzTargetFinder.java:90)
at com.code_intelligence.jazzer.driver.FuzzTargetFinder.findFuzzTarget(FuzzTargetFinder.java:68)
at com.code_intelligence.jazzer.driver.Driver.start(Driver.java:152)
at com.code_intelligence.jazzer.Jazzer.start(Jazzer.java:115)
at com.code_intelligence.jazzer.Jazzer.main(Jazzer.java:74)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Logger
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 11 more
@jonathanmetzman, thanks so much for fixing the issue. I am afk. I will take care of the failing fuzzer as the first thing next week Monday, the 26th.
Op ma 19 aug 2024 om 19:15 schreef jonathanmetzman @.***
Trying to clone the corpus doesn't work due to insufficient rights:
$ gsutil cp -r \ gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_Log4jFuzzer \ gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j-core-fuzz-test-PatternLayoutFuzzer Copying gs://log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_Log4jFuzzer/000aeaa9dfccf425541f6b7cbcc4d10977e6052b [Content-Type=application/octet-stream]... AccessDeniedException: 403 vol@.ci does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).
Neither the "Upload testcase" screen is allowing me to choose one of the new targets, e.g., log4j-core-fuzz-test-PatternLayoutFuzzer:
This issue is fixed now. I see some of the fuzzers running: gs:// log4j2-corpus.clusterfuzz-external.appspot.com/libFuzzer/log4j2_log4j-layout-template-json-fuzz-test-JsonTemplateLayoutCodecFuzzer/
others seem to be exceptioning. E.g. log4j-core-fuzz-test-PatternLayoutFuzzer
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended INFO: Loaded 261 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks INFO: Loaded 5 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks INFO: Loaded 2 hooks from com.code_intelligence.jazzer.sanitizers.ClojureLangHooks INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection INFO: Loaded 46 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection INFO: Loaded 48 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks INFO: Loaded 12 hooks from com.code_intelligence.jazzer.sanitizers.ScriptEngineInjection INFO: Loaded 3 hooks from com.code_intelligence.jazzer.sanitizers.ServerSideRequestForgery INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection INFO: Loaded 6 hooks from com.code_intelligence.jazzer.sanitizers.XPathInjection INFO: Instrumented org.apache.logging.log4j.core.fuzz.PatternLayoutFuzzer (took 185 ms, size +3%) Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger at java.base/java.lang.Class.getDeclaredMethods0(Native Method) at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3325) at java.base/java.lang.Class.getMethodsRecursive(Class.java:3466) at java.base/java.lang.Class.getMethod0(Class.java:3452) at java.base/java.lang.Class.getMethod(Class.java:2199) at com.code_intelligence.jazzer.driver.ReflectionUtils.targetPublicStaticMethod(ReflectionUtils.java:27) at com.code_intelligence.jazzer.driver.FuzzTargetFinder.findFuzzTargetByMethodName(FuzzTargetFinder.java:90) at com.code_intelligence.jazzer.driver.FuzzTargetFinder.findFuzzTarget(FuzzTargetFinder.java:68) at com.code_intelligence.jazzer.driver.Driver.start(Driver.java:152) at com.code_intelligence.jazzer.Jazzer.start(Jazzer.java:115) at com.code_intelligence.jazzer.Jazzer.main(Jazzer.java:74) Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Logger at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 11 more
— Reply to this email directly, view it on GitHub https://github.com/google/oss-fuzz/issues/12349#issuecomment-2297052454, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAARTSPV5A6I22XFHZHYTEDZSIRZTAVCNFSM6AAAAABMRCNOCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGA2TENBVGQ . You are receiving this because you authored the thread.Message ID: @.***>
vy
commented
1 month ago I've read both Debugging and Reproducing pages, and followed below steps:
# Verify fuzzer build
export PROJECT_NAME=log4j2
python infra/helper.py build_image $PROJECT_NAME
python infra/helper.py build_fuzzers --sanitizer address --engine libfuzzer --architecture x86_64 $PROJECT_NAME
python infra/helper.py check_build --sanitizer address --engine libfuzzer --architecture x86_64 $PROJECT_NAME log4j-core-fuzz-test-PatternLayoutFuzzer
# Run the fuzzer
python infra/helper.py run_fuzzer --sanitizer address --engine libfuzzer --architecture x86_64 $PROJECT_NAME log4j-core-fuzz-test-PatternLayoutFuzzer
# [Fuzzer runs successfully]Yet, I am not able to reproduce the runtime failures. @jonathanmetzman, minding helping with the issue, please?
vy
commented
1 month ago I've figured out the issue: I was storing dot-prefixed (in particular, .m2 for the Maven cache) files in $OUT (i.e., /out). Apparently ClusterFuzz isn't picking up dot-prefixed files in /out. Renaming them to something else fixed the problem.
Now I have other problems: Some(!) fuzzers, sometimes(!) use Java 15, even though they should have been using Java 17 installed by the Dockerfile. I will first try to troubleshoot this problem myself, and if I can't have much progress, then I will create another ticket.
log4j2fuzzing is recently revamped in #12304. Even though there are successful builds from 2024-08-13 and 2024-08-14, I am not able to see the associated runs of the newly added fuzzers in the ClusterFuzz web interface. Could you help me to troubleshoot the issue, please?