Closed FabrizioSandri closed 2 years ago
FabrizioSandri
commented
2 years ago I think that the problem is due to the fact that DeepState creates the output file once the test is completed. This is a problem since a segmentation fault in a test harness might force DeepState to crash before it gets a chance to write the input data. I am working on this.
FabrizioSandri
commented
2 years ago I've finally found a solution to this issue. The solution is to create two tests inside the same harness: one test will only generate fresh input data (without running the function being tested), while the second test will execute the function being tested taking as input the outputs of the first test(the generator).
In fact, when we run the first passing using the function deepstate harness analyze pkg, all we want at the end of this step is a Test Harness with its built binary file and a set of inputs; so in this first pass the execution of the tested function is not necessary. The execution based on the inputs should be done by the deepstate_harness_analyze_pkg function.
Let's see an example to clarify this solution. The test harness generated for the rcpp_read_out_of_bound function of the testSAN package will look like the one in the next fragment of code, where we have a generator that generates the inputs, without actually executing the function rcpp_read_out_of_bound and a runner that takes the inputs of the generator.
#include <fstream>
#include <RInside.h>
#include <iostream>
#include <RcppDeepState.h>
#include <qs.h>
#include <DeepState.hpp>
RInside Rinstance;
int rcpp_read_out_of_bound(int rbound);
TEST(testSAN, generator){
std::cout << "input starts" << std::endl;
IntegerVector rbound(1);
rbound[0] = RcppDeepState_int();
qs::c_qsave(rbound,"/home/fabri/test/testHarness/RcppDeepState/inst/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/inputs/rbound.qs",
"high", "zstd", 1, 15, true, 1);
std::cout << "rbound values: "<< rbound << std::endl;
std::cout << "input ends" << std::endl;
}
TEST(testSAN, runner){
std::cout << "input starts" << std::endl;
IntegerVector rbound(1);
rbound[0] = RcppDeepState_int();
std::cout << "rbound values: "<< rbound << std::endl;
std::cout << "input ends" << std::endl;
try{
rcpp_read_out_of_bound(rbound[0]);
}
catch(Rcpp::exception& e){
std::cout<<"Exception Handled"<<std::endl;
}
}We can begin by first creating the inputs using the following command, where rcpp_read_out_of_bound_DeepState_TestHarness is the compiled test harness.
$ ./rcpp_read_out_of_bound_DeepState_TestHarness --fuzz
--fuzz_save_passing --output_test_dir ./rcpp_read_out_of_bound_output
--input_which_test testSAN_generatorAs you can see the parameter --input_which_test is used to specify that during this phase, the generator test is run. After this phase, the rcpp_read_out_of_bound_output will contain some inputs files. We can analyze one of them using Valgrind in the following way:
$ valgrind ./rcpp_read_out_of_bound_DeepState_TestHarness --input_test_file
./rcpp_read_out_of_bound_output/00009f0507a2dbd3bdf0ca6b7ebb8049149a4904.pass
--input_which_test testSAN_runnerIn this case the parameter --input_which_test is used to specify that we want to run the runner test.
I'm going to implement this change in the pull request #6 .
tdhock
commented
2 years ago sounds great thanks
FabrizioSandri
commented
2 years ago The solution for this issue has been implemented in the pull request #6. More precisely in the following commits: 1ebc81c, 71edb6a, 05a7ed8
While writing the GitHub Action for the pull request #6 I came up again to the strange Segmentation fault error mentioned in the issue #2. I had assumed that the lack of debug symbols was to cause for the issue, however it doesn't appear that this is the case. The segmentation fault that I am referring to, occurs for the
rcpp_use_after_deallocatefunction in the testSAN package.Steps to reproduce
First of all I ran the test harness compilation procedure
deepstate_harness_compile_runand it succesfully generated the compiled test harness. The first execution however leaves thercpp_use_after_deallocate_outputempty. So I decided to manually run the harness with the same seed that generated the segmentation fault(5) to understand the problem.It appears that the segmentation fault occurs before deepstate can generate the output file. If I run the same program above using valgrind, the output folder is instead filled with a test file.
The inverse problem
@tdhock, I found this old conversation in the issue https://github.com/akhikolla/RcppDeepState/issues/62 about using Vaglrind in the first steps, after the harness compilation. Based on this discussion, I discovered that when the seed is set to 2, if valgrind is used to run the test, no output file is produced.
As you can see running this test standalone generates an output file.
Instead, if Valgrind is used, the program is aborted ad stated in the message
cannot throw exceptions and so is aborting instead. Sorry.