search

microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.52k stars 283 forks source link

Set origin in SequenceTracker when replaying a sequence from a log file #844

Closed wilbaker closed 6 months ago

wilbaker commented 6 months ago

With the trace database enabled, RESTler was throwing exceptions when attempting to use a replay file because origin was not being set.

This PR sets the origin to "replay", the same used by replay_sequence here:

https://github.com/microsoft/restler-fuzzer/blob/a2a41d728f974951a9853327a5db0b81e2035bfa/restler/engine/core/sequences.py#L805

Additionally, this PR adds a new test_trace_database_replay_file test to help keep this scenario healthy, and it updates test_trace_database_minimal to validate the origin in trace database requests.

I confirmed the new test fails without the fix in driver.py

marina-p commented 6 months ago

Thank you for the fix!

wilbaker commented 6 months ago

Looks like unit test is failing with the latest update. I am unable to repro locally so I'll be adding some additional logging to help debug the failure.

wilbaker commented 6 months ago

Looks like unit test is failing with the latest update. I am unable to repro locally so I'll be adding some additional logging to help debug the failure.

Unit test is passing again now with the additional logging, if there is anything I should do to reduce the chance of this test being flaky, please let me know. Thanks!

marina-p commented 6 months ago

To track down the unit test flakiness, one thing to try is to add a loop around the test logic and let it run 10-20 times locally to see if you can repro with the additional logging.

In reply to: 1865027029

wilbaker commented 6 months ago

To track down the unit test flakiness, one thing to try is to add a loop around the test logic and let it run 10-20 times locally to see if you can repro with the additional logging.

In reply to: 1865027029

Thanks for this suggestion! I believe I have found a different bug related to a race condition and I'm working on a fix now (will push to this same PR).

The issue is that when using a replay file, RESTler does not wait for the trace DB thread, it has a hard sys.exit(0) as soon as the sequence has finished.

Every so often this results in the last message being missed in the trace DB

wilbaker commented 6 months ago

To track down the unit test flakiness, one thing to try is to add a loop around the test logic and let it run 10-20 times locally to see if you can repro with the additional logging. In reply to: 1865027029

Thanks for this suggestion! I believe I have found a different bug related to a race condition and I'm working on a fix now (will push to this same PR).

The issue is that when using a replay file, RESTler does not wait for the trace DB thread, it has a hard sys.exit(0) as soon as the sequence has finished.

Every so often this results in the last message being missed in the trace DB

Just pushed a fix for this issue. Without the fix, I could consistently hit the race when I would run the unit test 200 times. I am now seeing no failures after running the test 800 times.

marina-p commented 6 months ago

Ah - thanks for tracking it down.

In reply to: 1865169155