search

CodeIntelligenceTesting / jazzer

Coverage-guided, in-process fuzzing for the JVM
https://code-intelligence.com
Other
1.03k stars 137 forks source link

The merge operation could fail when there's channel exception for coverage files #802

Open hadi88 opened 1 year ago

hadi88 commented 1 year ago

We've observed that CoverageIdStrategy.obtainFirstId fails during the merge process with FileLockInterruptionException, ClosedByInterruptException, and other channels exceptions. Handling such exceptions could significantly improve robustness of the merge operation.

Unfortunately, I don't have a reproducer for this issue.

fmeum commented 1 year ago

It would be really helpful to have a reproducer for this, but I can see that being difficult to come up with.

We have observed rare inconsistencies in the shared coverage ID file in the past that resulted in a crash, but never managed to identify the problem. I haven't seen cases of these exceptions being thrown yet, but will take another look at the code and check whether we can handle them (more) gracefully.

fmeum commented 1 year ago

@hadi88 Do you have stack traces that you could share?

I can attempt a speculative fix that retries obtaining the lock in case of interrupts, but I would first like to confirm that this has a chance to fix the issue you are seeing.

hadi88 commented 1 year ago

Initially, I observed a check failure in commitIdCount:

Caused by: java.lang.IllegalStateException: Check failed.
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.commitIdCount(CoverageIdStrategy.kt:175)
    ... 34 more

This exception masks the root cause. When I changed the check failure into an if-condition, I observed the following two stack traces:

ERROR: Coverage IDs are out of sync 
com.code_intelligence.jazzer.agent.CoverageIdException: Failed to synchronize coverage IDs
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:167)
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.withIdForClass(CoverageIdStrategy.kt:90)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.instrument(RuntimeInstrumentor.kt:215)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transformInternal(RuntimeInstrumentor.kt:185)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:96)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:161)
    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
    at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3139)
    at java.base/java.lang.Class.getConstructor0(Class.java:3344)
    at java.base/java.lang.Class.getConstructor(Class.java:2151)
    at io.netty.channel.ReflectiveChannelFactory.<init>(ReflectiveChannelFactory.java:34)
    at io.netty.bootstrap.AbstractBootstrap.channel(AbstractBootstrap.java:110)
    ...
Caused by: java.nio.channels.ClosedByInterruptException
    at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
    at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162)
    at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:228)
    at com.code_intelligence.jazzer.agent.CoverageIdStrategyKt.readFully(CoverageIdStrategy.kt:213)
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:123)
    ... 43 more
ERROR: Coverage IDs are out of sync 
com.code_intelligence.jazzer.agent.CoverageIdException: Failed to synchronize coverage IDs
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:167)
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.withIdForClass(CoverageIdStrategy.kt:90)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.instrument(RuntimeInstrumentor.kt:215)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transformInternal(RuntimeInstrumentor.kt:185)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:96)
    at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:161)
    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
    ...
Caused by: java.nio.channels.FileLockInterruptionException
    at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138)
    at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1065)
    at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:119)
    ... 46 more
fmeum commented 1 year ago

It would be interesting to learn what causes these exceptions. Is your fuzz test multi-threaded? Does your JVM support concurrent class loading?

I am asking as I haven't seen these failures anywhere else yet and handling them properly isn't trivial. With more information, I might be able to come up with a reproducer.

ghost commented 8 months ago

@hadi88 - I wrote to you about the other issue, same here. Ping me so we can look at it more closely together? david[dot]merian [at] code-intelligence[dot]com