Target binary is not instrumented

[retpoline]

Encountered a similar problem as in https://github.com/richo/afl-ruby/issues/1 while just running the example harness as described in the README. The patch was applied, which is why AFL continues instead of dies via fatal(), but it still does with fork server handshake failure. AFL_SKIP_BIN_CHECK=1 makes no difference for this one nor does upping the memory limit as suggested.

Is this a Ruby gem yet? I noticed "make" failed with the included Makefile, looks like the version is wrong (0.0.0 instead of 0.0.3) if nothing else.

Is there supposed to be a "make install" somewhere for the lib? I don't see how this was made to work for anything else outside of the example harness, maybe. We appreciate the work on the project here for sure, it just seems like the installation/instructions aren't portable enough to make the project easy to build and use.

Thanks for your time and support! Big hopes for this project to make security in the Ruby work better!

@richo @pje

[richo]

Hey there,

Thanks for opening the issue. Could you post the failure you got with the included makefile?

I will have a crack at running it in a fresh environment to see if it's just cruft from working on it that makes it work for me.

[retpoline]

Sure, thanks.

So building the extension works fine, but "make" or "make install" in the root directory fails.

$ make
gem build afl.gemspec
  Successfully built RubyGem
  Name: afl
  Version: 0.0.3
  File: afl-0.0.3.gem
gem uninstall --ignore-dependencies afl
Gem 'afl' is not installed
gem install --verbose afl-0.0.0.gem
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=afl-0.0.0.gem
200 OK
ERROR:  Could not find a valid gem 'afl-0.0.0.gem' (>= 0) in any repository
GET https://api.rubygems.org/latest_specs.4.8.gz
200 OK
GET https://api.rubygems.org/prerelease_specs.4.8.gz
200 OK
Makefile:15: recipe for target 'install' failed
make: *** [install] Error 2

If this was expected (which, it probably shouldn't be, perhaps part of the cruft ;), and fuzzing should just work anyways after patching, then AFL still complains about memory limit (often not really a memory issue, but a general sign of not being able to run the harness properly, eg. Ruby parser itself fails to run the harness).

That being said, if you're still able to get it working today, could you post the exact commands and harness used so if nothing else I can try and repro it exactly to rule out any weird environment error?

Ubuntu 18 is the OS, fwiw.

[richo]

Oh, I see. Yeah, changing the VERSION at the top of the makefile should make that work, although since it's literally just installing the gem.

I feel like at some point that used to also build AFL.. or was at least supposed to. The memory limit is something you will almost certainly have to bump, likely to a fairly extreme value.

I think on balance it's more likely to be bitrot than your environment, no promises but I will attempt to get a chance to look into it more closely.

[retpoline]

Ah, I'll try that. Looking forward to hearing back when you have a moment.

Thanks for taking a look!

[richo]

@inthenets ok! So I had a chance to have a quick look this morning. A few things:

Yeah that make file doesn't work unless you update the version. I'll push that now and make a note to add a test for that or try to make it more durable.

I didn't have to screw with memory limit in order to find the crash in the example. I more or less just did whatever the readme told me to, and it worked ok. As a note:

[-] Looks like the target binary is not instrumented! The fuzzer depends on           
    compile-time instrumentation to isolate interesting test cases while 
    mutating the input data. For more information, and for tips on how to       
    instrument binaries, please see docs/README.

afl-fuzz will spit out this warning. I forget what triggers it (I think maybe it's just a dry shot run writing not writing anything to the bitmap) which is expected, since a dry shot run of the ruby interpreter never actually boots the VM.

The instance I tested on is pretty slow and I was only seeing 80 execs/s which is pretty bad, but it only took about 8 seconds to find the crash. I remember working on the forkserver and persistent mode but honestly don't remember if they ever got stable enough to ship. I no longer have access to the computer I wrote this on (I did this while I was at Stripe) but I can probably go spelunking at some point and see if I can get it working again.

Anyway, tl;dr once you patch the makefile to make installing the Gem work, it should Just Work™ or at least did for me. I just invoked ~/AFL/afl-fuzz -i example/work/input -o example/work/output -- /usr/bin/ruby example/harness.rb

Please feel free to show me what's not working for you though and I will be happy to help.

[retpoline]

Yes, the target binary not instrumented is understood as normal as the patch just makes AFL not exit when such "funny conditions" occur to the fuzzer in this environment. Kudos on that too.

I followed the steps exactly from the README, but still get the memory limit fatal error. I suspect ruby doesn't like the harness, but it's unclear.

Could you provide the a list of the exact commands you just used to build and test from start to finish fuzzing /w crash found it so it's clear i'm doing the same, ruling out execution error?

That would be super helpful for my retesting.

[richo]

Is the fatal error emitted by AFL or by your environment? (Are you in docker or something?)

I can give you the exact commands when I'm back to my personal computer, but I don't think there's anything super special in there, I don't want to tell you this is exact in case it's not but it was more or less:

# cd afl-ruby
make && sudo make install
# cd afl
git apply < ~/afl-ruby/afl.patch
git commit -m 'afl-ruby patch'
make
# cd afl-ruby
~/afl/afl-fuzz -i example/work/input -o example/work/output -- /usr/bin/ruby example/harness.rb

I didn't touch memory limits, just used whatever defaults were on the instance when I got it. If you can post the actual memory error we can narrow it down to AFL being upset or something to do with your environment, I think. Can you also post the output of ulimit -a ?

[retpoline]

Fatal by AFL re: memory limit. The ulimits are default Ubuntu, nothing special there, I really think it may just be some step out of wack which is annoying Ruby upon execution.

But yes, when you convenient for you, I'd prefer to double confirm things with the exact commands as that would be very useful (and an easy guide/script for others in the future who may be having issues).

[richo]

I don't think ruby cares about your memory limits, and definitely doesn't with this harness which is tiny.

git clone git://github.com/richo/afl-ruby.git                                                                                                                                               
cd afl-ruby/                                                                                                                                                                                
make                                                                                                                                                                                        
sudo make install                                                                                                                                                                           
cd ..                                                                                                                                                                                       
git clone git://github.com/google/AFL.git                                                                                                                                                   
cd AFL/                                                                                                                                                                                     
git apply ../afl-ruby/afl-fuzz.c.patch                                                                                                                                                      
git commit -va                                                                                                                                                                              
make                                                                                                                                                                                        
cd ../afl-ruby/                                                                                                                                                                             
~/AFL/afl-fuzz -i example/work/input  -o example/work/output -- /usr/bin/ruby example/harness.rb                                                                                            
cat example/work/output/crashes/id\:000000\,sig\:10\,src\:000003\,op\:flip4\,pos\:3                                                                                                         

Here's my shell history for the session that I got everything working, nothing really special there. Can you post the exact error message?

[retpoline]

Oh, I didn't mean that ruby was having memory issues, but that that the memory limit error can be somewhat generically thrown if the interpreter for example can't run the harness correctly, or other errors the interpreter is having. AFL is a little funny in this way, but I guess it does it's best with the information it has.

Really good to see the commands spelled out, as when first going over the README, a few things weren't exactly clear such as whether to get AFL from the original website, from google github, or another source. That and the make error can get you thrown off track pretty quickly.

But hooray, it's working now. It did throw a memory error limit at first, which was strange, but adjusting it with AFL got it working again. This did not work when testing previously (!) -- must have been something slightly off during build or the gem errors.

Thanks for making the changes to the gem versioning and updates! I'd suggest, if you also believe it would be useful for others, to add these concise instructions to the README as well.

[richo]

Gotcha. I'll update the readme to include those steps, for sure. Thanks for that.

That makes sense RE afl. I'll address that specifically but more or less any AFL will work, I'm sure if you go back far enough you bump into issues (not supporting the leader/follower stuff or whatever) but it's been a very long time since I used any new AFL features.

Thanks again, and good luck :)