Support replay mode with garbage collection

[marina-p]

Support replay mode with garbage collection

This change enables a new 'replay' mode option that replays requests from the trace database.

Changes include:

1. Refactor driver to consume sequences from different sources (with trace database another supported source of specific sequences, similar to the existing smoke test mode)

2. Generate 'replay blocks' as part of payload rendering. Replay blocks contain dynamic objects, but do not support custom payloads (this is future work).

3. Enables replay to be based on replay blocks

With this change, RESTler generates sequences from replay blocks if available, or from raw request/responses (similar to the existing replay mode) when they are not available.

The current version of replay is based on the rendered data sent at the time of original sequence rendering. This has several limitations, such as not being able to garbage-collect resources or plugging in custom payloads.

In particular, when a bug is reproduced in RESTler today, the same replay mechanism is used, which means that GC does not collect resources created while reproducing the bug. Moreover, re-running the same sequence without GC will not work if the resource has not been deleted if a resource-generating request has a unique ID parameter (e.g. PUT /resource/{resourceId}), causing false negative non-reproducible bugs.

This change addresses this issue by implementing replay blocks-based replay, which is invoked when reproducing bugs.

Note: the existing replay based on .replay.txt files will also be able to use this new mechanism if a grammar is provided, but this is not implemented as part of this change.

4. Added a setting to filter the origin during DB replay.

To only replay specific origins, add the following to the settings file: "replay": { "include_origins": ["main_driver", "InvalidValueChecker"] }

Testing:

• manual testing of demo_server replay as follows:

1) Run 'test' task and generate trace database

restler.exe test --grammar_file .\Compile\grammar.py --dictionary_file .\Compile\dict.json --host localhost --target_port 8888 --settings .\compile\engine_settings.json --no_ssl

Engine settings: { "use_trace_database": true, "trace_database": { "root_dir": "d:\demo_server\trace_databases", }, }

2) Run 'replay' task from above trace database

restler.exe replay --replay_log ./trace_data.ndjson --grammar_file .\Compile\grammar.py --dictionary_file .\Compile\dict.json --host localhost --target_port 8888 --settings .\compile\engine_settings.json --no_ssl

• updated unit tests

[wilbaker]

I have been doing some testing with the changes in this branch and noticed:

• GC is being skipped when replaying the ndjson from my smoke test
• Sometimes paths constructed by RESTler for requests have "null" in them instead of the expected string value (returned by the service in response to an earlier request). Example: /api/carts/null instead of /api/carts/wbfuzzstr_cart
• Only a subset of sequences are being run

Possibly related, I did see some messages like this in the main.txt of my replay:

Found a matching prefix of length 2 for request 4 with previous request -1 but that prefix was INVALID (root = -1) #Resolved

[wilbaker]

@marina-p with the latest changes I am seeing an engine failure when running the fuzz command:

2024-01-17 10:30:36.204: Initializing: Trace database.
2024-01-17 10:30:36.219: Initializing: Garbage collection after every test sequence. 
2024-01-17 10:30:36.236: Terminating garbage collection. Waiting for max 300 seconds. 
2024-01-17 10:30:36.236: Finishing writing to Trace Database. Waiting for max 10 seconds. 
Traceback (most recent call last):
  File "D:\scratch\advancedReplay\restler-bin\engine\engine\core\fuzzer.py", line 43, in run
    self._num_total_sequences = driver.generate_sequences(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\scratch\advancedReplay\restler-bin\engine\engine\core\driver.py", line 766, in generate_sequences
    if seq_constraints_by_generation[generation]:
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 1
``` #Resolved

[wilbaker]

After testing with the latest changes I've found that replay_blocks is not being populated for call sequences that RESTler sends when it decides to replay a sequence while fuzzing (e.g., after receiving a 500).

Would it be possible to include the replay_blocks in those sequences as well? #Resolved

[Yanqiulei]

How can I resolve the following error when using Restler for testing functionality? 2024-01-24 16:39:01.091: Connection error: 'Exception: [Errno 104] Connection reset by peer' #Closed

[wilbaker]

Hi @marina-p

I've been doing some additional testing and noticed that sometimes when my test service returns a 404, I see the following message logged in the network.testing file:

Parser exception: 'Error: all of the expected dynamic objects were not present in the response.'.

I'm not sure if this is related to these changes, but sharing it in case you think it might be.

Thanks! #Resolved

[wilbaker]

I noticed that when replaying sequences, the network.testing logs often have the Generation & Sequences logged twice, once at the start of the file and then again at the end of the file:

Generation-4: Rendering Sequence-1

    Request: 1 (Current combination: 0 / 2)
...
...

// Send and receive logs
...
...
Generation-4: Rendering Sequence-1

    Request: 1 (Current combination: 0 / 2)

The requests for the sequence are only sent once though (as expected) #Resolved

[marina-p]

Hello @Yanqiulei, I just noticed this comment is in a PR that is about to be closed. If you are still having this problem, could you please re-post it in an issue?

---

In reply to: 1907659288