Unexpected timeout when fuzzing

[singleghost2]

Hello! I use jackalope to fuzz my target binary. Executing my binary on the command line alone can execute and exit normally in less than 1 second, but when using jackalope, it will timeout, even if the -t option is set to a very long time, such as 10 seconds, it will also timeout. The log is as followings, is there any ideas about what is happening? I tried to add -trace_debug_events and -trace_basic_blocks, the output shows that same basic block set is executed over and over again and seems to never stop. I suspect that the issue may lie in the instrumentation module of TinyInst.

~/workspace/apple_fuzz/webaudio_fuzz(master*) »  sudo ~/softwares/Jackalope/build/Release/fuzzer -in corpus_in/test -out corpus_out/tmp -t 10000 -delivery file     -instrument_module AudioToolboxCore -instrument_module AudioCodecs  -stack_offset 0x1000 -covtype edge -cmp_coverage true -patch_return_addresses -trace_debug_events -- ./audio_dec @@
Fuzzer version 0.01
1 input files read
Running input sample corpus_in/test/timeout.caf
Debugger: Mach exception (5) @ address 0x119b96000
Debugger: Process created or attached
Debugger: Loaded module /usr/lib/dyld at 0x119b95000
Debugger: Loaded module audio_dec at 0x10cee8000
Debugger: Loaded module Foundation at 0x7fff2120e000
Debugger: Loaded module AudioToolbox at 0x7fff2cc31000
Debugger: Loaded module libSystem.B.dylib at 0x7fff2a5bf000
Debugger: Loaded module CoreFoundation at 0x7fff2045f000
Debugger: Loaded module libobjc.A.dylib at 0x7fff2027f000
Debugger: Loaded module CoreAutoLayout at 0x7fff2720a000
Debugger: Loaded module SystemConfiguration at 0x7fff20f22000
Debugger: Loaded module libz.1.dylib at 0x7fff2a4f7000
Debugger: Loaded module libfakelink.dylib at 0x7fff2a5c1000
Debugger: Loaded module libcompression.dylib at 0x7fff2a836000
Debugger: Loaded module CFNetwork at 0x7fff24745000
Debugger: Loaded module DiskArbitration at 0x7fff265dd000
Debugger: Loaded module libarchive.2.dylib at 0x7fff2a701000
Debugger: Loaded module libDiagnosticMessagesClient.dylib at 0x7fff264f2000
Debugger: Loaded module libicucore.A.dylib at 0x7fff225ce000
Debugger: Loaded module libxml2.2.dylib at 0x7fff27252000
Debugger: Loaded module CoreServices at 0x7fff2ff5d000
Debugger: Loaded module liblangid.dylib at 0x7fff29060000
Debugger: Loaded module IOKit at 0x7fff22b9e000
Debugger: Loaded module libCRFSuite.dylib at 0x7fff20fa0000
Debugger: Loaded module SoftLinking at 0x7fff2a5c5000
Debugger: Loaded module libc++abi.dylib at 0x7fff2039b000
Debugger: Loaded module liboah.dylib at 0x7fff2a58c000
Debugger: Loaded module libc++.1.dylib at 0x7fff20345000
Debugger: Loaded module libcache.dylib at 0x7fff2a5b9000
Debugger: Loaded module libcommonCrypto.dylib at 0x7fff2a575000
Debugger: Loaded module libcompiler_rt.dylib at 0x7fff2a59f000
Debugger: Loaded module libcopyfile.dylib at 0x7fff2a594000
Debugger: Loaded module libcorecrypto.dylib at 0x7fff2016f000
Debugger: Loaded module libdispatch.dylib at 0x7fff2023a000
Debugger: Loaded module libdyld.dylib at 0x7fff203ed000
Debugger: Loaded module libkeymgr.dylib at 0x7fff2a5b0000
Debugger: Loaded module liblaunch.dylib at 0x7fff2d9c8000
Debugger: Loaded module libmacho.dylib at 0x7fff2a553000
Debugger: Loaded module libquarantine.dylib at 0x7fff29d4b000
Debugger: Loaded module libremovefile.dylib at 0x7fff2a5ad000
Debugger: Loaded module libsystem_asl.dylib at 0x7fff24c51000
Debugger: Loaded module libsystem_blocks.dylib at 0x7fff2011f000
Debugger: Loaded module libsystem_c.dylib at 0x7fff202bc000
Debugger: Loaded module libsystem_collections.dylib at 0x7fff2a5a7000
Debugger: Loaded module libsystem_configuration.dylib at 0x7fff29051000
Debugger: Loaded module libsystem_containermanager.dylib at 0x7fff2833d000
Debugger: Loaded module libsystem_coreservices.dylib at 0x7fff2a2f6000
Debugger: Loaded module libsystem_darwin.dylib at 0x7fff2282e000
Debugger: Loaded module libsystem_dnssd.dylib at 0x7fff2a5b1000
Debugger: Loaded module libsystem_featureflags.dylib at 0x7fff202b9000
Debugger: Loaded module libsystem_info.dylib at 0x7fff20433000
Debugger: Loaded module libsystem_m.dylib at 0x7fff2a50a000
Debugger: Loaded module libsystem_malloc.dylib at 0x7fff2020d000
Debugger: Loaded module libsystem_networkextension.dylib at 0x7fff24be4000
Debugger: Loaded module libsystem_notify.dylib at 0x7fff22c4d000
Debugger: Loaded module libsystem_product_info_filter.dylib at 0x7fff2fe5f000
Debugger: Loaded module libsystem_sandbox.dylib at 0x7fff29055000
Debugger: Loaded module libsystem_secinit.dylib at 0x7fff2a5aa000
Debugger: Loaded module libsystem_kernel.dylib at 0x7fff203b1000
Debugger: Loaded module libsystem_platform.dylib at 0x7fff20429000
Debugger: Loaded module libsystem_pthread.dylib at 0x7fff203e1000
Debugger: Loaded module libsystem_symptoms.dylib at 0x7fff2632e000
Debugger: Loaded module libsystem_trace.dylib at 0x7fff20157000
Debugger: Loaded module libunwind.dylib at 0x7fff2a581000
Debugger: Loaded module libxpc.dylib at 0x7fff20121000
Debugger: Loaded module libbsm.0.dylib at 0x7fff29d72000
Debugger: Loaded module libnetwork.dylib at 0x7fff240be000
Debugger: Loaded module libpcap.A.dylib at 0x7fff2a5c6000
Debugger: Loaded module libdns_services.dylib at 0x7fff26327000
Debugger: Loaded module libcoretls_cfhelpers.dylib at 0x7fff2ac63000
Debugger: Loaded module Security at 0x7fff22276000
Debugger: Loaded module libapple_nghttp2.dylib at 0x7fff2a800000
Debugger: Loaded module libenergytrace.dylib at 0x7fff24bf3000
Debugger: Loaded module libkxld.dylib at 0x7fff2a559000
Debugger: Loaded module libsqlite3.dylib at 0x7fff25fa0000
Debugger: Loaded module libMobileGestalt.dylib at 0x7fff24bf4000
Debugger: Loaded module AppleFSCompression at 0x7fff2a2d9000
Debugger: Loaded module libcoretls.dylib at 0x7fff29d5a000
Debugger: Loaded module libpam.2.dylib at 0x7fff2a831000
Debugger: Loaded module libxar.1.dylib at 0x7fff2ad61000
Debugger: Loaded module AppleSystemInfo at 0x7fff2905c000
Debugger: Loaded module IOMobileFramebuffer at 0x7fff2999c000
Debugger: Loaded module IOSurface at 0x7fff2834e000
Debugger: Loaded module libbz2.1.0.dylib at 0x7fff2a2e9000
Debugger: Loaded module liblzma.5.dylib at 0x7fff2ac4a000
Debugger: Loaded module libiconv.2.dylib at 0x7fff2a5fe000
Debugger: Loaded module libcharset.1.dylib at 0x7fff2a552000
Debugger: Loaded module FSEvents at 0x7fff271d5000
Debugger: Loaded module CarbonCore at 0x7fff22838000
Debugger: Loaded module Metadata at 0x7fff26542000
Debugger: Loaded module OSServices at 0x7fff2a2fb000
Debugger: Loaded module SearchKit at 0x7fff2a773000
Debugger: Loaded module AE at 0x7fff262b2000
Debugger: Loaded module LaunchServices at 0x7fff208fd000
Debugger: Loaded module DictionaryServices at 0x7fff2abfa000
Debugger: Loaded module SharedFileList at 0x7fff271e2000
Debugger: Loaded module libCheckFix.dylib at 0x7fff29d4e000
Debugger: Loaded module TCC at 0x7fff24c68000
Debugger: Loaded module CoreNLP at 0x7fff29062000
Debugger: Loaded module MetadataUtilities at 0x7fff264f5000
Debugger: Loaded module libmecabra.dylib at 0x7fff20fd5000
Debugger: Loaded module MLCompute at 0x7fff2a058000
Debugger: Loaded module Accelerate at 0x7fff30119000
Debugger: Loaded module libmecab.dylib at 0x7fff29d83000
Debugger: Loaded module libgermantok.dylib at 0x7fff29dcd000
Debugger: Loaded module libThaiTokenizer.dylib at 0x7fff2a7db000
Debugger: Loaded module libChineseTokenizer.dylib at 0x7fff2ad9c000
Debugger: Loaded module MetalPerformanceShaders at 0x7fff2a82f000
Debugger: Loaded module Metal at 0x7fff2836a000
Debugger: Loaded module vImage at 0x7fff265e4000
Debugger: Loaded module vecLib at 0x7fff2ff37000
Debugger: Loaded module libvMisc.dylib at 0x7fff2ada7000
Debugger: Loaded module libvDSP.dylib at 0x7fff297b7000
Debugger: Loaded module libBLAS.dylib at 0x7fff20c08000
Debugger: Loaded module libLAPACK.dylib at 0x7fff2a85c000
Debugger: Loaded module libLinearAlgebra.dylib at 0x7fff29dd3000
Debugger: Loaded module libSparseBLAS.dylib at 0x7fff2a818000
Debugger: Loaded module libQuadrature.dylib at 0x7fff2a856000
Debugger: Loaded module libBNNS.dylib at 0x7fff2910e000
Debugger: Loaded module libSparse.dylib at 0x7fff20eb3000
Debugger: Loaded module MPSCore at 0x7fff28fea000
Debugger: Loaded module MPSImage at 0x7fff2a248000
Debugger: Loaded module MPSNeuralNetwork at 0x7fff29de9000
Debugger: Loaded module MPSMatrix at 0x7fff2a1ba000
Debugger: Loaded module MPSRayIntersector at 0x7fff2a008000
Debugger: Loaded module MPSNDArray at 0x7fff2a1f1000
Debugger: Loaded module MetalTools at 0x7fff20b33000
Debugger: Loaded module AggregateDictionary at 0x7fff2905a000
Debugger: Loaded module CoreAnalytics at 0x7fff264c2000
Debugger: Loaded module AppleSauce at 0x7fff2a7dd000
Debugger: Loaded module IOAccelerator at 0x7fff28360000
Debugger: Loaded module libCoreFSCache.dylib at 0x7fff6babc000
Debugger: Loaded module LanguageModeling at 0x7fff2156d000
Debugger: Loaded module CoreEmoji at 0x7fff2998a000
Debugger: Loaded module LinguisticData at 0x7fff29107000
Debugger: Loaded module Lexicon at 0x7fff20e65000
Debugger: Loaded module libcmph.dylib at 0x7fff2a6ef000
Debugger: Loaded module CFOpenDirectory at 0x7fff271b5000
Debugger: Loaded module OpenDirectory at 0x7fff271a5000
Debugger: Loaded module APFS at 0x7fff2ac65000
Debugger: Loaded module SecurityFoundation at 0x7fff29cb0000
Debugger: Loaded module libutil.dylib at 0x7fff2ad6f000
Debugger: Loaded module libapp_launch_measurement.dylib at 0x7fff27207000
Debugger: Loaded module CoreServicesStore at 0x7fff22b63000
Debugger: Loaded module ServiceManagement at 0x7fff29d46000
Debugger: Loaded module libxslt.1.dylib at 0x7fff2ad73000
Debugger: Loaded module BackgroundTaskManagement at 0x7fff29d3c000
Debugger: Loaded module AudioToolboxCore at 0x7fff2178d000
Debugger: Loaded module AudioSession at 0x7fff2cd66000
Debugger: Loaded module caulk at 0x7fff2848e000
Debugger: Loaded module CoreAudio at 0x7fff21be4000
Debugger: Loaded module libAudioToolboxUtility.dylib at 0x7fff2bc4c000
Debugger: Loaded module ProtocolBuffer at 0x7fff25f8a000
Debugger: Loaded module AppServerSupport at 0x7fff2b6ce000
Debugger: Loaded module perfdata at 0x7fff2d76b000
Debugger: Loaded module AssertionServices at 0x7fff29c9f000
Debugger: Loaded module SystemPolicy at 0x7fff3cc17000
Debugger: Loaded module libIOReport.dylib at 0x7fff2aefd000
Debugger: Loaded module libSMC.dylib at 0x7fff2cfa7000
Debugger: Loaded module BaseBoard at 0x7fff261e9000
Debugger: Loaded module RunningBoardServices at 0x7fff26269000
Debugger: Loaded module PersistentConnection at 0x7fff2b232000
Debugger: Loaded module CoreGraphics at 0x7fff24fe7000
Debugger: Loaded module ImageIO at 0x7fff28a31000
Debugger: Loaded module CommonUtilities at 0x7fff261d1000
Debugger: Loaded module Bom at 0x7fff2b94b000
Debugger: Loaded module SkyLight at 0x7fff24c81000
Debugger: Loaded module libFontParser.dylib at 0x7fff28596000
Debugger: Loaded module WatchdogClient at 0x7fff2bab2000
Debugger: Loaded module CoreDisplay at 0x7fff21656000
Debugger: Loaded module CoreMedia at 0x7fff284ab000
Debugger: Loaded module CoreVideo at 0x7fff27335000
Debugger: Loaded module MultitouchSupport at 0x7fff2bab6000
Debugger: Loaded module QuartzCore at 0x7fff26c4c000
Debugger: Loaded module VideoToolbox at 0x7fff2baed000
Debugger: Loaded module GPUWrangler at 0x7fff2bd24000
Debugger: Loaded module IOPresentment at 0x7fff2bd07000
Debugger: Loaded module DSExternalDisplay at 0x7fff2bd30000
Debugger: Loaded module CMCaptureCore at 0x7fff2bd90000
Debugger: Loaded module libspindump.dylib at 0x7fff2b6e8000
Debugger: Loaded module ColorSync at 0x7fff25671000
Debugger: Loaded module libate.dylib at 0x7fff2ae65000
Debugger: Loaded module libRadiance.dylib at 0x7fff2bd2c000
Debugger: Loaded module libJPEG.dylib at 0x7fff2bd36000
Debugger: Loaded module libPng.dylib at 0x7fff2bc80000
Debugger: Loaded module libTIFF.dylib at 0x7fff2bca7000
Debugger: Loaded module libGIF.dylib at 0x7fff2bd8b000
Debugger: Loaded module libJP2.dylib at 0x7fff2b9d2000
Debugger: Loaded module libexpat.1.dylib at 0x7fff2b0ee000
Debugger: Loaded module AppleJPEG at 0x7fff2b988000
Debugger: Loaded module OpenGL at 0x7fff6bac8000
Debugger: Loaded module libGLU.dylib at 0x7fff6bb18000
Debugger: Loaded module libGFXShared.dylib at 0x7fff6badb000
Debugger: Loaded module libGL.dylib at 0x7fff6bce9000
Debugger: Loaded module libGLImage.dylib at 0x7fff6bae4000
Debugger: Loaded module libCVMSPluginSupport.dylib at 0x7fff6bad8000
Debugger: Loaded module libCoreVMClient.dylib at 0x7fff6bac3000
Debugger: Loaded module CoreImage at 0x7fff28c6e000
Debugger: Loaded module CoreText at 0x7fff219fe000
Debugger: Loaded module OpenCL at 0x7fff6d133000
Debugger: Loaded module GraphVisualizer at 0x7fff2b25b000
Debugger: Loaded module FaceCore at 0x7fff2b26a000
Debugger: Loaded module OTSVG at 0x7fff2b686000
Debugger: Loaded module libFontRegistry.dylib at 0x7fff26f2a000
Debugger: Loaded module libhvf.dylib at 0x7fff2b6d5000
Debugger: Loaded module AppleVA at 0x7fff2b09a000
Debugger: Loaded module libmis.dylib at 0x7fff3d574000
Debugger: Loaded module libAudioStatistics.dylib at 0x7fff2cdcc000
Debugger: Loaded module MediaExperience at 0x7fff2b193000
Debugger: Loaded module libSessionUtility.dylib at 0x7fff2cbff000
Debugger: Loaded module libperfcheck.dylib at 0x7fff2d777000
Debugger: Loaded module AudioResourceArbitration at 0x7fff2d116000
Debugger: Loaded module CoreData at 0x7fff25b6a000
Debugger: Loaded module libSimplifiedChineseConverter.dylib at 0x7fff5cef7000
Debugger: Process entrypoint reached
Instrumented module AudioToolboxCore, code size: 2560000
Debugger: Loaded module AudioCodecs at 0x10cf0b000
Instrumented module AudioCodecs, code size: 6176768

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 1

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0

Total execs: 1
Unique samples: 0 (0 discarded)
Crashes: 0 (0 unique)
Hangs: 0
Offsets: 0
Execs/s: 0
Debugger: Process exit

[ifratric]

Hi and sorry for a somewhat late reply, I just returned from my vacation. I see you closed the issue in the meantime but I still wanted to offer some thoughts, in case it helps:

Note that -patch_return_addresses has a significant performance impact so it's quite possible that something that takes <1s naively takes >10s with -patch_return_addresses. You can test if that's the case for you by trying to run without that flag, though you might encounter issues later in the fuzzing session if your target relies on C++ / Objective C exceptions.

We are currently working on a solution that will remove the need for -patch_return_addresses and allow fuzzing of such targets at near native speeds, but it's not ready just yet.

[singleghost2]

Thansk for you reply! I absolutely need patch_return_addresses now because if not, I will receive libc++abi: terminating with uncaught exception of type CAXException error. I use perisistent mode afterwards and see a performance improve to 0.8/s, hope that your work can bring the efficiency of tinyinst to a higher level. Thanks!

[singleghost2]

And I found that the increase in the number of threads is not proportional to the increase in fuzz speed. As the number of threads increases, the increase in fuzz speed becomes more and more slow. Too many threads may even lead to a decrease in fuzz speed. Any ideas about this?

[ifratric]

For multiple threads - this is likely because of work done by the OS kernel - depending on how much time is spent in the target process vs. how much time is spent in the kernel, you will see a greater or lesser benefit with the number of threads. Unfortunately, -patch_return_addresses also introduces a lot of kernel overhead (requiring context switching often etc.) so it also negatively affects this (but on the positive side, once we remove it, you should also see improvement in parallel fuzzing).

One thing you could experiment with is, instead of having one fuzzer process with large -nthreads, you could have several fuzzer processes which synchronize over Jackalope server (see Jackalope -start_server flag). Whether you'll get any benefit from this depends on how the operating system kernel is implemented (how locking mechanisms work etc.), but it's worth experimenting with.