Local clusterfuzz reports bot as dead after a few minutes/hours

[Manouchehri]

I'm following the heartbleed tutorial and while clusterfuzz does manage to find a couple bugs as expected, my bot stops fuzzing.

Is there some configuration to tell clusterfuzz to continue fuzzing until I say stop, or is this a bug? I couldn't find anything in the documentation or GH issues about it.

I'm running 69844f7b6ca1783ee4b520c303a6a45c294461c4.

http://localhost:9000/bots

my-bot
fuzz libFuzzer libfuzzer_asan_linux_openssl
ends Mar 8, 10:34 AM
Last Ping - Mar 8, 4:34 AM

Main application log snippet:

| INFO     2020-03-08 17:19:43,165 module.py:865] cron-service: "GET /triage HTTP/1.1" 200 -
| INFO     2020-03-08 17:19:53,506 instance.py:294] Instance PID: 30818
| /home/clusterfuzz/clusterfuzz/src/appengine/third_party/urllib3/contrib/appengine.py:115: AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets. To use sockets directly instead of URLFetch see https://urllib3.readthedocs.io/en/latest/reference/urllib3.contrib.html.
|   AppEnginePlatformWarning)
| INFO     2020-03-08 17:19:53,802 driver.py:124] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
| INFO     2020-03-08 17:19:53,815 driver.py:124] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
| INFO     2020-03-08 17:19:54,065 module.py:865] default: "GET /_ah/start HTTP/1.1" 404 52
| INFO     2020-03-08 17:19:54,181 module.py:865] default: "GET /bots HTTP/1.1" 200 986230
| INFO     2020-03-08 17:19:54,332 module.py:865] default: "GET /favicon/manifest.json?v=A0RYylnML6 HTTP/1.1" 304 -
| INFO     2020-03-08 17:20:02,796 logs.py:344] Processing testcase 4573968371548160.
| INFO     2020-03-08 17:20:02,807 logs.py:344] Processing testcase 5488762045857792.
| INFO     2020-03-08 17:20:02,819 logs.py:344] Processing testcase 6051711999279104.
| INFO     2020-03-08 17:20:02,893 module.py:865] cron-service: "GET /cleanup HTTP/1.1" 200 -
| INFO     2020-03-08 17:20:43,406 module.py:865] cron-service: "GET /triage HTTP/1.1" 200 -
| INFO     2020-03-08 17:21:33,108 logs.py:344] Processing testcase 4573968371548160.
| INFO     2020-03-08 17:21:33,129 logs.py:344] Processing testcase 5488762045857792.
| INFO     2020-03-08 17:21:33,143 logs.py:344] Processing testcase 6051711999279104.
| INFO     2020-03-08 17:21:33,212 module.py:865] cron-service: "GET /cleanup HTTP/1.1" 200 -
| INFO     2020-03-08 17:21:43,611 module.py:865] cron-service: "GET /triage HTTP/1.1" 200 -
| INFO     2020-03-08 17:22:43,812 module.py:865] cron-service: "GET /triage HTTP/1.1" 200 -
| INFO     2020-03-08 17:23:03,361 logs.py:344] Processing testcase 4573968371548160.
| INFO     2020-03-08 17:23:03,377 logs.py:344] Processing testcase 5488762045857792.
| INFO     2020-03-08 17:23:03,389 logs.py:344] Processing testcase 6051711999279104.
| INFO     2020-03-08 17:23:03,469 module.py:865] cron-service: "GET /cleanup HTTP/1.1" 200 -

Bot log snippet:

2020-03-08 08:34:09,223 - run_bot - INFO - Finished processing crashes.
2020-03-08 08:34:09,224 - run_bot - INFO - New crashes: 0, known crashes: 4, processed groups: [<bot.tasks.fuzz_task.CrashGroup object at 0x7fdc28de1190>]
2020-03-08 08:34:09,293 - run_bot - INFO - Clearing defunct process 10937.
2020-03-08 08:34:09,294 - run_bot - INFO - Clearing defunct process 10944.
2020-03-08 08:34:09,414 - run_bot - INFO - Using local source, skipping source code update.
2020-03-08 08:34:09,414 - run_bot - INFO - Running platform initialization scripts.
2020-03-08 08:34:09,921 - run_bot - INFO - Completed running platform initialization scripts.
2020-03-08 08:34:10,169 - run_bot - INFO - Executing command 'fuzz libFuzzer libfuzzer_asan_linux_openssl'
2020-03-08 08:34:10,286 - run_bot - INFO - Setting up fuzzer and data bundles.
2020-03-08 08:34:10,307 - run_bot - INFO - Retrieving custom binary build r1.
2020-03-08 08:34:10,307 - run_bot - INFO - Build already exists.
2020-03-08 08:34:10,312 - run_bot - INFO - Picked fuzz target handshake-fuzzer for fuzzing.
2020-03-08 08:34:10,312 - run_bot - INFO - Setup application path.
2020-03-08 08:34:10,315 - run_bot - INFO - Checking for bad build.

[inferno-chromium]

from bot log snippet, bot seems to be working as expected and still running. Do you see it dead on /bots page ?

[Manouchehri]

@inferno-chromium Yeah, I agree that nothing looks wrong in the logs. It feels like this is a PEBKAC issue with a misunderstanding of how jobs are scheduled.

My CPU usage goes down to pretty much zero and there's no handshake-fuzzer processes to be seen (ps aux). The Last Ping stops updating as well.

I just did a fresh install and went back to v1.8.0 to see if the same behaviour occurs, will upload a screenshot once that finishes.

[Manouchehri]

The fuzzers continue to work with v1.8.0. Not sure what broke since then.

[inferno-chromium]

Cannot reproduce.