"create_once" causes not 100% coverage for all APIs

[qihongchen]

Description

I need to test an internal REPL API, there are 9 APIs in it, and the resource REPL can only be created 5 times for its parent resource. Since there's no way to control how many instance of a resource to create except "create_once", "create_once" is used in my test. But this caused following coverage report:

Using python: 'python.exe' (Python 3.8.2) Request coverage (successful / total): 8 / 9 Attempted requests: 8 / 9 No bugs were found.

I checked logs, found the API to create REPL appeared in log file "network.preprocessing.704.1.txt", all other 8 APIs appeared in network log file ("network.testing.5560.1.txt"). Because all 9 APIs are invoked at least once in the test, the coverage report should show "9 / 9", i.e., 100%.

Steps to reproduce

Create a set of test APIs, one of the API to create an instance of the resource, and other APIs can check resource details, or use the resource do something. Set up "create_once" in the engine_settings.json, then run "restler.ext test ..." command, you'll see the coverage result is not 100%.

Expected results

expected coverage result is 9 / 9 (assume 9 APIs)

Actual results

the coverage result was 8 / 9.

Environment details

Windows 11, Python 3.8.2, dotnet 7.0.402, restler 9.2.2

[marina-p]

Hello @qihongchen, Thanks for reporting this.

The inconsistency is due to the fact that the intent of the coverage data is to show how much of the API operations will be fuzzed. 'create_once' is a feature to save time pre-creating resources outside of RESTler, which will not be fuzzed. So, the coverage data shown is correct as intended.

I understand that what you are looking for is to reflect that you were able to successfully exercise all APIs in Test mode. If we add a new line to the testing summary and output that will account for the 'create_once' resources, would that be helpful for your use case? i.e.:

Using python: 'python.exe' (Python 3.8.2) Request coverage (successful / total): 8 / 9 Attempted requests: 8 / 9 Create once requests: 1 / 9 No bugs were found.

Also, the run_gc_after_every_sequence engine setting may help avoid having to add this resource to 'create_once'.

Thanks,

Marina