Another major problem our project faces is how to do evaluation properly.
I think there're two parts of evaluation: precision/correctness and performance. In our case, performance evaluation is relatively less important but easier to do. Precision/correctness evaluation is more important. However, since we don't have a dataset with manually checked cases to test our analysis, I'm not sure how to do it properly. Currently, we are just manually reading the code, and compare it with the path found by analysis one by one, which is painful and not scalable. If we keep doing that, our evaluation will end up with a very small set of manually checked cases in the end.
Another major problem our project faces is how to do evaluation properly.
I think there're two parts of evaluation: precision/correctness and performance. In our case, performance evaluation is relatively less important but easier to do. Precision/correctness evaluation is more important. However, since we don't have a dataset with manually checked cases to test our analysis, I'm not sure how to do it properly. Currently, we are just manually reading the code, and compare it with the path found by analysis one by one, which is painful and not scalable. If we keep doing that, our evaluation will end up with a very small set of manually checked cases in the end.
@tianyin Do you have any suggestions?