Help with Python Sanitizer Issue

[sg3-141-592]

I've been trying to increase coverage for the urllib3 library by fuzzing it's HTTP requests functionality (see fuzz_requests.py). I've tried to do this through spawning a Python HTTPServer onto a separate thread, and requesting data off it.

This is running fine on local introspector runs (i.e. infra/helper.py introspector urllib3), but is consistently timing out running on oss-fuzz.

#179: cov: 0 ft: 0 corp: 0 exec/s 0 oom/timeout/crash: 0/178/0 time: 6253s job: 179 dft_time: 0

And occasionally erroring out libFuzzer_urllib3_fuzz_requests/libfuzzer_ubsan_urllib3 - 2023-05-03 - 22:18:26:474500.log

Component revisions (build r202305030615):
Urllib3: 6351614959b6599fe53312223c972daba75a671f

Bot name: oss-fuzz-linux-zone4-host-wbx4-3
Return code: 1

Command: /mnt/scratch0/clusterfuzz/resources/platform/linux/unshare -c -n /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds_urllib3_fbf55b28bec3b301768e00a81ba78154ba96fc5d/revisions/fuzz_requests -timeout=25 -rss_limit_mb=2560 -artifact_prefix=/mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/ -max_total_time=6300 -print_final_stats=1 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases-disk/temp-383/new /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/urllib3_fuzz_requests
Time ran: 64.70086646080017

INFO: Using preloaded libfuzzer
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1841444727
INFO:        0 files found in /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases-disk/temp-383/new
INFO:      242 files found in /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/urllib3_fuzz_requests
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 677238 bytes

 === Uncaught Python exception: ===
MaxRetryError: HTTPConnectionPool(host='localhost', port=9935): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79cd5aa9fb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
Traceback (most recent call last):
  File "fuzz_requests.py", line 124, in TestOneInput
  File "urllib3/_request_methods.py", line 118, in request
  File "urllib3/_request_methods.py", line 217, in request_encode_body
  File "urllib3/poolmanager.py", line 433, in urlopen
  File "urllib3/connectionpool.py", line 874, in urlopen
  File "urllib3/connectionpool.py", line 874, in urlopen
  File "urllib3/connectionpool.py", line 874, in urlopen
  File "urllib3/connectionpool.py", line 844, in urlopen
  File "urllib3/util/retry.py", line 515, in increment
MaxRetryError: HTTPConnectionPool(host='localhost', port=9935): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79cd5aa9fb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

 === Timeout: 26s elapsed, timeout=25s ===
INFO: Instrumenting 3826 functions...
Thread 0x000079cd56e8a700 (most recent call first):
  File "selectors.py", line 415 in select
  File "socketserver.py", line 232 in serve_forever
  File "fuzz_requests.py", line 70 in run
  File "threading.py", line 870 in run
  File "threading.py", line 932 in _bootstrap_inner
  File "threading.py", line 890 in _bootstrap

Current thread 0x000079cd5b5b7780 (most recent call first):
  File "threading.py", line 1388 in _shutdown
ALARM: working on the last Unit for 26 seconds
       and the timeout value is 25 (use -timeout=N to change)
MS: 0 ; base unit: 0000000000000000000000000000000000000000

Note, there's messages around the code instrumenting while the exception is firing which seems to indicate an ordering issue.

I'm stuck at this point. I've tried improving the performance of the fuzzer, moving the thread spawning into the TestOneInput method and providing some retry logic for getting a port binding. Is there anything else I should be trying or anything fundamental I'm missing around how fuzzing works that might be a root cause?

[jonathanmetzman]

I don't have any ideas here sorry.

[DaveLak]

I believe this issue can be closed as https://github.com/google/oss-fuzz/pull/11120 and https://github.com/google/oss-fuzz/pull/11161 appear to have made this a non-issue?