Shorten CMakeLists, create C++ script for differential testing

SVF-tools / Test-Suite

PTABen: Micro-benchmark Suite for Pointer Analysis

70 stars 38 forks source link

Shorten CMakeLists, create C++ script for differential testing #39

Closed lukekoko closed 3 years ago

lukekoko commented 3 years ago

Now using for loops in CMakeLists (reduced lines to 86 compared to 1000+)
Removed generate_cmake_test.sh as not needed anymore
Created C++ script to do differential testing

lukekoko commented 3 years ago

Some testcases fail

yuleisui commented 3 years ago

Is this for Andersen or for flow-sensitive analysis? How about the differential analysis for flow-sensitivity?

lukekoko commented 3 years ago

The differential analysis for fspta-vfspta and ander-nander are in. fspta-vfspta have no errors. ander-nander 2 test cases have errors.

yuleisui commented 3 years ago

The differential analysis for fspta-vfspta and ander-nander are in. fspta-vfspta have no errors. ander-nander 2 test cases have errors.

I saw 28 errors?

lukekoko commented 3 years ago

The other errors are from the normal tests with one option

lukekoko commented 3 years ago

For example this is a test of wpa -ander -stat=false that fails

lukekoko commented 3 years ago

The tests that are getting errors do not exist in the test cases in master branch. For example, there are no mem_leak tests in master. Do you want these removed?

yuleisui commented 3 years ago

For example this is a test of wpa -ander -stat=false that fails

Strange.. It was correct for the recent commit in main repo. Can you take a look?

https://travis-ci.org/github/SVF-tools/SVF/jobs/769922964#L10761-L10771

lukekoko commented 3 years ago

For example this is a test of wpa -ander -stat=false that fails

Strange.. It was correct for the recent commit in main repo. Can you take a look?

https://travis-ci.org/github/SVF-tools/SVF/jobs/769922964#L10761-L10771

For this one the test is missing -ff-eq-base=false. I'll add this in

yuleisui commented 3 years ago

For example this is a test of wpa -ander -stat=false that fails

Strange.. It was correct for the recent commit in main repo. Can you take a look? https://travis-ci.org/github/SVF-tools/SVF/jobs/769922964#L10761-L10771

For this one the test is missing -ff-eq-base=false. I'll add this in

yuleisui commented 3 years ago

For example this is a test of wpa -ander -stat=false that fails

Strange.. It was correct for the recent commit in main repo. Can you take a look? https://travis-ci.org/github/SVF-tools/SVF/jobs/769922964#L10761-L10771

For this one the test is missing -ff-eq-base=false. I'll add this in

OK

How about the other failed tests? Can you list them here? We can check one by one.

lukekoko commented 3 years ago

yuleisui commented 3 years ago

Can we exclude these failed tests and add the successful ones for now?

We will fix the failed CPP ones so that we can add them all into this repo.

yuleisui commented 3 years ago

Maybe we create a folder to store those CPP_Failed cases. For memory leak cases, we may not include any of them for now.

lukekoko commented 3 years ago

Maybe we create a folder to store those CPP_Failed cases. For memory leak cases, we may not include any of them for now.

Ok, I'll move the failed tests to a failed folder and exclude the memory leak tests

yuleisui commented 3 years ago

I assume that after this patch, SVF will still pass all the test cases in CMakeList?

lukekoko commented 3 years ago

Failed test files have been moved to failed folder. All tests are now passing