search

MozillaSecurity / funfuzz

A collection of fuzzers in a harness for testing the SpiderMonkey JavaScript engine.
Mozilla Public License 2.0
630 stars 115 forks source link

Unable to find required entries in FuzzManager. Duplicate detection via sigcache will not work... #215

Closed Implementist closed 5 years ago

Implementist commented 5 years ago

I installed FuzzManager via using pip install FuzzManager and finished configuring funfuzz . The root folder of my Ubuntu server is /home/nisl/ and I created a file as /home/nisl/.fuzzmanagerconf which contains content like:

[Main]
sigdir = /home/nisl/sigcache/
serverhost = 127.0.0.1
serverport = 8000
serverproto = https
tool = jsfunfuzz

When I run python -u funfuzz/loop_bot.py -b "--random" --target-time 28800 | tee ~/log-loop_botPy.txt in /home/nisl/funfuzz/src/ I always get an error like:

Unable to find required entries in FuzzManager. Duplicate detection via sigcache will not work...
/tmp/tmp962g_8cyfuzzbot
ERROR:boto:Caught exception reading instance data
Traceback (most recent call last):
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/http/client.py", line 936, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/socket.py", line 722, in create_connection
    raise err
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
socket.timeout: timed out

Could anybody who tells me how to fix it?

nth10sd commented 5 years ago

Does this still occur if you install funfuzz requirements (FuzzManager included) using:

pip install --upgrade -r requirements.txt

? Ideally you should create a virtualenv from Python 3 via -m venv.

Implementist commented 5 years ago

I tried pip install --upgrade -r requirements.txt and the output becomes:

Unable to find required entries in FuzzManager. Duplicate detection via sigcache will not work...
/tmp/tmptdcfdx0ffuzzbot
Traceback (most recent call last):
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/nisl/funfuzz/src/funfuzz/bot.py", line 242, in <module>
    main()
  File "/home/nisl/funfuzz/src/funfuzz/bot.py", line 115, in main
    build_info = ensureBuild(options)
  File "/home/nisl/funfuzz/src/funfuzz/bot.py", line 187, in ensureBuild
    compile_shell.obtainShell(cshell, updateLatestTxt=updateLatestTxt)
  File "/home/nisl/funfuzz/src/funfuzz/js/compile_shell.py", line 632, in obtainShell
    raise OSError("Found a cached shell that failed compilation...")
OSError: Found a cached shell that failed compilation...
Something went wrong when calling: ['/home/nisl/anaconda3/envs/py3.6_env/bin/python', '-u', '-m', 'funfuzz.bot', '-b', '--random', '--target-time', '28800']
CalledProcessError(1, ['/home/nisl/anaconda3/envs/py3.6_env/bin/python', '-u', '-m', 'funfuzz.bot', '-b', '--random', '--target-time', '28800'])

and I execute these commands under a python 3.6 environment which been created by anaconda3.

nth10sd commented 5 years ago

Please first remove your existing ~/shell-cache directory (or rename it) which had cached the previous failed operation, then try again.

Implementist commented 5 years ago

I did that and get the error again. But when I open the new, only one file in shell-cache here is the content:

Configuration of mozilla-central rev f6df375b8698 failed with the following output:
sh: 0: Can't open /home/nisl/trees/mozilla-central/js/src/configure

Caught exception CalledProcessError(127, ['sh', '/home/nisl/trees/mozilla-central/js/src/configure', '--target=i686-pc-linux', '--enable-more-deterministic', '--with-ccache', '--enable-gczeal', '--enable-debug-symbols', '--disable-tests']) (Command '['sh', '/home/nisl/trees/mozilla-central/js/src/configure', '--target=i686-pc-linux', '--enable-more-deterministic', '--with-ccache', '--enable-gczeal', '--enable-debug-symbols', '--disable-tests']' returned non-zero exit status 127.)
Backtrace:
Traceback (most recent call last):
  File "/home/nisl/funfuzz/src/funfuzz/js/compile_shell.py", line 672, in obtainShell
    cfgJsCompile(shell)
  File "/home/nisl/funfuzz/src/funfuzz/js/compile_shell.py", line 334, in cfgJsCompile
    cfgBin(shell)
  File "/home/nisl/funfuzz/src/funfuzz/js/compile_shell.py", line 529, in cfgBin
    stdout=subprocess.PIPE).stdout.decode("utf-8", errors="replace")
  File "/home/nisl/anaconda3/envs/py3.6_env/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['sh', '/home/nisl/trees/mozilla-central/js/src/configure', '--target=i686-pc-linux', '--enable-more-deterministic', '--with-ccache', '--enable-gczeal', '--enable-debug-symbols', '--disable-tests']' returned non-zero exit status 127.

I opened /home/nisl/trees/mozilla-central/js/src/ but found that there have not a file named configure. The most similar one is configure.in.

nth10sd commented 5 years ago

At least we're making progress.

I have not tried with anaconda before. Do you mind creating a virtualenv using the -m venv method, then repeating the steps?

Assuming you're on 18.04 Bionic, you could try:

python3 -m venv ~/venv-funfuzz
pushd ~/funfuzz && ~/venv-funfuzz/bin/pip install --upgrade -r requirements.txt && popd
# This should test compilation only, as opposed to compilation + fuzzing for funfuzz.bot
~/venv-funfuzz/bin/python -m funfuzz.js.compile_shell -b "--enable-debug --enable-more-deterministic"
Implementist commented 5 years ago

OK, I will try it and report the result later.

Implementist commented 5 years ago

Mr. Kwong, Because of the server I used before can not run venv method correctly, I created a virtual machine of Ubuntu 16.04. With both conda and python -m venv I configured funfuzz two times again. But I got the same Error Can't open /home/nisl/trees/mozilla-central/js/src/configure. Is it necessary to run these commands under Ubuntu 18.04?

nth10sd commented 5 years ago

Yes, they are needed to compile the shell. I'm not sure why configure cannot be accessed now. Perhaps a permissions issue? They should all be run under the ubuntu user.

I won't really support 16.04 going forward, everything should be proceeding towards 18.04.

Implementist commented 5 years ago

Thanks for your help up to now. I will try it again under a Ubuntu 18.04 emvironment and report the result later.

Implementist commented 5 years ago

Mr. Kwong, I found out that antoconf should be installed and run autoconf in /home/username/trees/mozilla-central/js/src to generate configure file manually. Then although boto problem persists, compilation works normally. And after few hours of compilation I got output like:

/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r1 |  11.0s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r2 |   0.5s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r3 |  10.4s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r4 |   0.5s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r5 |   0.5s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r6 |   2.0s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r7 |   2.1s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r8 |  12.1s | 0 | fine
/tmp/tmpe9rsx2mqloop0/w994-cj-initial-r9 |  12.3s | 0 | fine

It seems I finished the configuration of funfuzz rightly. But I still have a question about boto: Is it necessary to create a ~/.boto file and what is the content for it?

nth10sd commented 5 years ago

Thanks for checking! Yes, the build prerequisites should first be installed, and that includes autoconf 2.13.

As for .boto, that would be for Amazon S3 build caching support. It is not strictly needed since the harness caches on the local ~/shell-cache directory as well.

Since funfuzz seems to now be running properly, I'm closing this issue.