search

microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.52k stars 283 forks source link

RESTler ran forever: "Exhausted collection..." loop prevented RESTler from honoring --time_budget #885

Closed wilbaker closed 2 months ago

wilbaker commented 2 months ago

Description

In a recent run of RESTler, the remaining_time_budget check started getting skipped because ExhaustSeqCollectionException kept getting thrown in a loop.

Because RESTler never made it far enough to check remaining_time_budget here:

https://github.com/microsoft/restler-fuzzer/blob/7503a3434a34c9cdf9e2732d47d45e182c6505fd/restler/engine/core/sequences.py#L668

timeout_reached was never being set to true in driver.py here:

https://github.com/microsoft/restler-fuzzer/blob/7503a3434a34c9cdf9e2732d47d45e182c6505fd/restler/engine/core/driver.py#L840

And RESTler never set should_stop to true here:

https://github.com/microsoft/restler-fuzzer/blob/7503a3434a34c9cdf9e2732d47d45e182c6505fd/restler/engine/core/driver.py#L878

The following was logged over and over until RESTler was killed:

Testing completed -- below are the final stats:

Setting fuzzing schemes: random-walk
2024-04-09 13:22:24.000: Going to fuzz a set with 47 requests
2024-04-09 13:22:24.000: Request-0: Value Combinations: 4
2024-04-09 13:22:24.000: Request-1: Value Combinations: 4
2024-04-09 13:22:24.000: Request-2: Value Combinations: 4
2024-04-09 13:22:24.000: Request-3: Value Combinations: 4
2024-04-09 13:22:24.000: Request-4: Value Combinations: 4
2024-04-09 13:22:24.000: Request-5: Value Combinations: 4
2024-04-09 13:22:24.000: Request-6: Value Combinations: 8
2024-04-09 13:22:24.000: Request-7: Value Combinations: 4
2024-04-09 13:22:24.000: Request-8: Value Combinations: 4
2024-04-09 13:22:24.000: Request-9: Value Combinations: 4
2024-04-09 13:22:24.000: Request-10: Value Combinations: 4
2024-04-09 13:22:24.000: Request-11: Value Combinations: 4
2024-04-09 13:22:24.000: Request-12: Value Combinations: 4
2024-04-09 13:22:24.000: Request-13: Value Combinations: 4
2024-04-09 13:22:24.000: Request-14: Value Combinations: 2
2024-04-09 13:22:24.000: Request-15: Value Combinations: 4
2024-04-09 13:22:24.000: Request-16: Value Combinations: 4
2024-04-09 13:22:24.000: Request-17: Value Combinations: 4
2024-04-09 13:22:24.000: Request-18: Value Combinations: 4
2024-04-09 13:22:24.000: Request-19: Value Combinations: 4
2024-04-09 13:22:24.000: Request-20: Value Combinations: 4
2024-04-09 13:22:24.000: Request-21: Value Combinations: 4
2024-04-09 13:22:24.000: Request-22: Value Combinations: 4
2024-04-09 13:22:24.000: Request-23: Value Combinations: 4
2024-04-09 13:22:24.000: Request-24: Value Combinations: 4
2024-04-09 13:22:24.000: Request-25: Value Combinations: 4
2024-04-09 13:22:24.000: Request-26: Value Combinations: 4
2024-04-09 13:22:24.000: Request-27: Value Combinations: 4
2024-04-09 13:22:24.000: Request-28: Value Combinations: 4
2024-04-09 13:22:24.000: Request-29: Value Combinations: 4
2024-04-09 13:22:24.000: Request-30: Value Combinations: 4
2024-04-09 13:22:24.000: Request-31: Value Combinations: 4
2024-04-09 13:22:24.000: Request-32: Value Combinations: 4
2024-04-09 13:22:24.000: Request-33: Value Combinations: 4
2024-04-09 13:22:24.000: Request-34: Value Combinations: 4
2024-04-09 13:22:24.000: Request-35: Value Combinations: 4
2024-04-09 13:22:24.000: Request-36: Value Combinations: 4
2024-04-09 13:22:24.000: Request-37: Value Combinations: 4
2024-04-09 13:22:24.000: Request-38: Value Combinations: 4
2024-04-09 13:22:24.000: Request-39: Value Combinations: 4
2024-04-09 13:22:24.000: Request-40: Value Combinations: 10
2024-04-09 13:22:24.000: Request-41: Value Combinations: 4
2024-04-09 13:22:24.000: Request-42: Value Combinations: 4
2024-04-09 13:22:24.000: Request-43: Value Combinations: 4
2024-04-09 13:22:24.000: Request-44: Value Combinations: 4
2024-04-09 13:22:24.000: Request-45: Value Combinations: 4
2024-04-09 13:22:24.000: Request-46: Value Combinations: 4
2024-04-09 13:22:24.000: Avg. Value Combinations per Request: 4.170212765957447
2024-04-09 13:22:24.000: Median Value Combinations per Request: 4
2024-04-09 13:22:24.000: Min Value Combinations per Request: 2
2024-04-09 13:22:24.000: Max Value Combinations per Request: 10
2024-04-09 13:22:24.000: Total dependencies: 68

2024-04-09 13:22:24.016: Generation: 1 / Sequences Collection Size: 1 
(After random-walk Extend)

Exhausted collection...
2024-04-09 13:26:36.350: Generation: 1 / Sequences Collection Size: 0 
(After random-walk Render)

2024-04-09 13:26:36.350: Final Swagger spec coverage: 17 / 47
2024-04-09 13:26:36.350: Rendered requests: 17 / 47
2024-04-09 13:26:36.350: Rendered requests with "valid" status codes: 17 / 17
2024-04-09 13:26:36.350: Num fully valid requests (no resource creation failures): 17
2024-04-09 13:26:36.350: Num requests not rendered due to invalid sequence re-renders: 0
2024-04-09 13:26:36.350: Num invalid requests caused by failed resource creations: 0
2024-04-09 13:26:36.350: Total Creations of Dyn Objects: 56
2024-04-09 13:26:36.350: Total Requests Sent: {'gc': 113, 'main_driver': 431}
2024-04-09 13:26:36.350: Bug Buckets: {'main_driver_501': 1}

Testing completed -- below are the final stats:

Setting fuzzing schemes: random-walk

Well after --time_budget was exceeded.

Steps to reproduce

I've only seen this once so far, and I'm not familiar enough about the implementation details of sequence rendering to understand the preconditions that triggered this issue.

RESTler was using the random-walk strategy in case that's relevant:

restler.exe fuzz --time_budget 3.8963747535833333 --search_strategy random-walk --disable_checkers useafterfree,invaliddynamicobject,leakagerule,resourcehierarchy,payloadbody,examples,invalidvalue --grammar_file "Compile\grammar.py" --dictionary_file "dict.json" --settings "engine_settings.json"

Expected results

I expected that Timed out... would be logged after the time budget was exceeded.

Actual results

RESTler kept running forever and logging "Exhausted collection..."

Environment details

RESTler version 9.2.4