search

microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.62k stars 303 forks source link

Set origin in SequenceTracker when replaying a sequence from a log file #844

Closed wilbaker closed 11 months ago

wilbaker commented 11 months ago

With the trace database enabled, RESTler was throwing exceptions when attempting to use a replay file because origin was not being set.

This PR sets the origin to "replay", the same used by replay_sequence here:

https://github.com/microsoft/restler-fuzzer/blob/a2a41d728f974951a9853327a5db0b81e2035bfa/restler/engine/core/sequences.py#L805

Additionally, this PR adds a new test_trace_database_replay_file test to help keep this scenario healthy, and it updates test_trace_database_minimal to validate the origin in trace database requests.

I confirmed the new test fails without the fix in driver.py

marina-p commented 11 months ago

Thank you for the fix!

wilbaker commented 11 months ago

Looks like unit test is failing with the latest update. I am unable to repro locally so I'll be adding some additional logging to help debug the failure.

wilbaker commented 11 months ago

Looks like unit test is failing with the latest update. I am unable to repro locally so I'll be adding some additional logging to help debug the failure.

Unit test is passing again now with the additional logging, if there is anything I should do to reduce the chance of this test being flaky, please let me know. Thanks!

marina-p commented 11 months ago

To track down the unit test flakiness, one thing to try is to add a loop around the test logic and let it run 10-20 times locally to see if you can repro with the additional logging.

In reply to: 1865027029

wilbaker commented 11 months ago

To track down the unit test flakiness, one thing to try is to add a loop around the test logic and let it run 10-20 times locally to see if you can repro with the additional logging.

In reply to: 1865027029

Thanks for this suggestion! I believe I have found a different bug related to a race condition and I'm working on a fix now (will push to this same PR).

The issue is that when using a replay file, RESTler does not wait for the trace DB thread, it has a hard sys.exit(0) as soon as the sequence has finished.

Every so often this results in the last message being missed in the trace DB

wilbaker commented 11 months ago

To track down the unit test flakiness, one thing to try is to add a loop around the test logic and let it run 10-20 times locally to see if you can repro with the additional logging. In reply to: 1865027029

Thanks for this suggestion! I believe I have found a different bug related to a race condition and I'm working on a fix now (will push to this same PR).

The issue is that when using a replay file, RESTler does not wait for the trace DB thread, it has a hard sys.exit(0) as soon as the sequence has finished.

Every so often this results in the last message being missed in the trace DB

Just pushed a fix for this issue. Without the fix, I could consistently hit the race when I would run the unit test 200 times. I am now seeing no failures after running the test 800 times.

marina-p commented 11 months ago

Ah - thanks for tracking it down.

In reply to: 1865169155