search

SVF-tools / Test-Suite

PTABen: Micro-benchmark Suite for Pointer Analysis
70 stars 38 forks source link

tests failed #3

Closed bitcalc closed 4 years ago

bitcalc commented 4 years ago

Hi,

I'm new to SVF and after I've set it up I ran some tests. To my surprise, it reported lots of failures. I listed them below. I'm not sure if they are such as expected. But since they are not marked as EXPECTED FAIL:, I would assume they should probably be fixed. My SVF commit is d2652172a50539c5146f004ddc5d47329a657080 and the PTABen commit is def08dd444a63393427a573b8939a20d4d4fd913. The only script I used was 'testwpa.sh'.

basic_c_tests/struct-twoflds.c @@@analyzing basic_c_tests/struct-twoflds.c with testwpa.sh [AndersenWPA] Checking NOALIAS FAIL :NOALIAS check <id:231, id:233> at (ln: 33 fl: basic_c_tests/struct-twoflds.c) FAIL :NOALIAS check <id:217, id:218> at (ln: 25 fl: basic_c_tests/struct-twoflds.c)

fs_tests/test-su.c @@@analyzing fs_tests/test-su.c with testwpa.sh FAIL :NOALIAS check <id:231, id:233> at (ln: 12 fl: fs_tests/test-su.c) FAIL :NOALIAS check <id:226, id:228> at (ln: 11 fl: fs_tests/test-su.c)

fs_tests/pcycle1.c @@@analyzing fs_tests/pcycle1.c with testwpa.sh FAIL :NOALIAS check <id:230, id:231> at (ln: 14 fl: fs_tests/pcycle1.c)

fs_tests/array_alias_2.c @@@analyzing fs_tests/array_alias_2.c with testwpa.sh FAIL :NOALIAS check <id:281, id:285> at (ln: 32 fl: fs_tests/array_alias_2.c) FAIL :NOALIAS check <id:273, id:277> at (ln: 31 fl: fs_tests/array_alias_2.c)

fs_tests/global_2.c @@@analyzing fs_tests/global_2.c with testwpa.sh FAIL :NOALIAS check <id:222, id:225> at (ln: 18 fl: fs_tests/global_2.c)

fs_tests/strong_update.c @@@analyzing fs_tests/strong_update.c with testwpa.sh FAIL :NOALIAS check <id:236, id:237> at (ln: 14 fl: fs_tests/strong_update.c) fs_tests/global_1.c @@@analyzing fs_tests/global_1.c with testwpa.sh FAIL :NOALIAS check <id:216, id:219> at (ln: 13 fl: fs_tests/global_1.c)

fs_tests/array_alias_1.c @@@analyzing fs_tests/array_alias_1.c with testwpa.sh FAIL :NOALIAS check <id:228, id:232> at (ln: 25 fl: fs_tests/array_alias_1.c) FAIL :NOALIAS check <id:219, id:223> at (ln: 24 fl: fs_tests/array_alias_1.c)

fs_tests/struct_1.c @@@analyzing fs_tests/struct_1.c with testwpa.sh FAIL :NOALIAS check <id:230, id:233> at (ln: 22 fl: fs_tests/struct_1.c) FAIL :NOALIAS check <id:223, id:226> at (ln: 21 fl: fs_tests/struct_1.c)

cs_tests/recur0.c @@@analyzing cs_tests/recur0.c with testwpa.sh FAIL :NOALIAS check <id:221, id:213> at (ln: 11 fl: cs_tests/recur0.c)

cs_tests/cs18.c @@@analyzing cs_tests/cs18.c with testwpa.sh FAIL :NOALIAS check <id:243, id:236> at (ln: 21 fl: cs_tests/cs18.c) FAIL :NOALIAS check <id:234, id:236> at (ln: 17 fl: cs_tests/cs18.c)

cs_tests/recur3.c @@@analyzing cs_tests/recur3.c with testwpa.sh FAIL :NOALIAS check <id:236, id:224> at (ln: 23 fl: cs_tests/recur3.c) FAIL :NOALIAS check <id:226, id:228> at (ln: 17 fl: cs_tests/recur3.c)

cs_tests/cs17.c @@@analyzing cs_tests/cs17.c with testwpa.sh FAIL :NOALIAS check <id:297, id:299> at (ln: 32 fl: cs_tests/cs17.c) FAIL :NOALIAS check <id:292, id:294> at (ln: 31 fl: cs_tests/cs17.c) FAIL :NOALIAS check <id:274, id:275> at (ln: 26 fl: cs_tests/cs17.c)

cs_tests/cs3.c @@@analyzing cs_tests/cs3.c with testwpa.sh FAIL :NOALIAS check <id:287, id:289> at (ln: 34 fl: cs_tests/cs3.c) FAIL :NOALIAS check <id:273, id:275> at (ln: 31 fl: cs_tests/cs3.c)

@@@analyzing cs_tests/cs4.c with testwpa.sh FAIL :NOALIAS check <id:267, id:269> at (ln: 21 fl: cs_tests/cs4.c) FAIL :NOALIAS check <id:262, id:264> at (ln: 20 fl: cs_tests/cs4.c)

cs_tests/cs9.c @@@analyzing cs_tests/cs9.c with testwpa.sh FAIL :NOALIAS check <id:282, id:284> at (ln: 25 fl: cs_tests/cs9.c) FAIL :NOALIAS check <id:232, id:234> at (ln: 11 fl: cs_tests/cs9.c)

cs_tests/cs2.c @@@analyzing cs_tests/cs2.c with testwpa.sh FAIL :NOALIAS check <id:289, id:291> at (ln: 37 fl: cs_tests/cs2.c) FAIL :NOALIAS check <id:276, id:278> at (ln: 34 fl: cs_tests/cs2.c)

cs_tests/cs20.c @@@analyzing cs_tests/cs20.c with testwpa.sh FAIL :NOALIAS check <id:259, id:232> at (ln: 20 fl: cs_tests/cs20.c) FAIL :NOALIAS check <id:234, id:236> at (ln: 12 fl: cs_tests/cs20.c)

cs_tests/recur4.c @@@analyzing cs_tests/recur4.c with testwpa.sh FAIL :NOALIAS check <id:239, id:231> at (ln: 27 fl: cs_tests/recur4.c) FAIL :NOALIAS check <id:233, id:234> at (ln: 19 fl: cs_tests/recur4.c)

cs_tests/recur10.c @@@analyzing cs_tests/recur10.c with testwpa.sh FAIL :NOALIAS check <id:227, id:218> at (ln: 17 fl: cs_tests/recur10.c) FAIL :NOALIAS check <id:223, id:225> at (ln: 16 fl: cs_tests/recur10.c)

cs_tests/funcpoiner.c @@@analyzing cs_tests/funcpoiner.c with testwpa.sh FAIL :NOALIAS check <id:242, id:243> at (ln: 23 fl: cs_tests/funcpoiner.c) FAIL :NOALIAS check <id:232, id:233> at (ln: 19 fl: cs_tests/funcpoiner.c)

cs_tests/cs16.c @@@analyzing cs_tests/cs16.c with testwpa.sh FAIL :NOALIAS check <id:243, id:245> at (ln: 23 fl: cs_tests/cs16.c) FAIL :NOALIAS check <id:238, id:240> at (ln: 22 fl: cs_tests/cs16.c) FAIL :NOALIAS check <id:233, id:235> at (ln: 21 fl: cs_tests/cs16.c)

cs_tests/cs11.c @@@analyzing cs_tests/cs11.c with testwpa.sh FAIL :NOALIAS check <id:237, id:238> at (ln: 17 fl: cs_tests/cs11.c) FAIL :NOALIAS check <id:233, id:234> at (ln: 16 fl: cs_tests/cs11.c)

cs_tests/cs7.c @@@analyzing cs_tests/cs7.c with testwpa.sh FAIL :NOALIAS check <id:265, id:267> at (ln: 27 fl: cs_tests/cs7.c) FAIL :NOALIAS check <id:260, id:262> at (ln: 26 fl: cs_tests/cs7.c)

cs_tests/cs19.c @@@analyzing cs_tests/cs19.c with testwpa.sh FAIL :NOALIAS check <id:218, id:219> at (ln: 12 fl: cs_tests/cs19.c)

cs_tests/cs0.c @@@analyzing cs_tests/cs0.c with testwpa.sh FAIL :NOALIAS check <id:228, id:229> at (ln: 16 fl: cs_tests/cs0.c) FAIL :NOALIAS check <id:225, id:226> at (ln: 15 fl: cs_tests/cs0.c)

path_tests/path6.c @@@analyzing path_tests/path6.c with testwpa.sh FAIL :NOALIAS check <id:251, id:252> at (ln: 21 fl: path_tests/path6.c) FAIL :NOALIAS check <id:247, id:248> at (ln: 20 fl: path_tests/path6.c)

path_tests/path17.c @@@analyzing path_tests/path17.c with testwpa.sh FAIL :NOALIAS check <id:255, id:256> at (ln: 26 fl: path_tests/path17.c)

path_tests/path3.c @@@analyzing path_tests/path3.c with testwpa.sh FAIL :NOALIAS check <id:243, id:244> at (ln: 24 fl: path_tests/path3.c) FAIL :NOALIAS check <id:240, id:241> at (ln: 23 fl: path_tests/path3.c)

path_tests/path5.c @@@analyzing path_tests/path5.c with testwpa.sh FAIL :NOALIAS check <id:254, id:255> at (ln: 28 fl: path_tests/path5.c) FAIL :NOALIAS check <id:244, id:245> at (ln: 26 fl: path_tests/path5.c)

path_tests/path21.c @@@analyzing path_tests/path21.c with testwpa.sh FAIL :NOALIAS check <id:241, id:242> at (ln: 24 fl: path_tests/path21.c) FAIL :NOALIAS check <id:238, id:239> at (ln: 23 fl: path_tests/path21.c)

path_tests/path18.c @@@analyzing path_tests/path18.c with testwpa.sh FAIL :NOALIAS check <id:244, id:231> at (ln: 16 fl: path_tests/path18.c) FAIL :NOALIAS check <id:240, id:242> at (ln: 15 fl: path_tests/path18.c) FAIL :NOALIAS check <id:237, id:235> at (ln: 14 fl: path_tests/path18.c)

path_tests/path4.c @@@analyzing path_tests/path4.c with testwpa.sh FAIL :NOALIAS check <id:240, id:241> at (ln: 26 fl: path_tests/path4.c)

path_tests/path20.c @@@analyzing path_tests/path20.c with testwpa.sh FAIL :MUSTALIAS check <id:224, id:225> at (ln: 18 fl: path_tests/path20.c)

path_tests/path8.c @@@analyzing path_tests/path8.c with testwpa.sh FAIL :NOALIAS check <id:264, id:265> at (ln: 30 fl: path_tests/path8.c) FAIL :NOALIAS check <id:260, id:261> at (ln: 29 fl: path_tests/path8.c)

path_tests/path13.c @@@analyzing path_tests/path13.c with testwpa.sh FAIL :NOALIAS check <id:253, id:254> at (ln: 26 fl: path_tests/path13.c) FAIL :NOALIAS check <id:249, id:250> at (ln: 25 fl: path_tests/path13.c)

path_tests/path16.c @@@analyzing path_tests/path16.c with testwpa.sh FAIL :NOALIAS check <id:249, id:250> at (ln: 23 fl: path_tests/path16.c) FAIL :NOALIAS check <id:245, id:246> at (ln: 22 fl: path_tests/path16.c)

path_tests/path15.c @@@analyzing path_tests/path15.c with testwpa.sh FAIL :NOALIAS check <id:224, id:225> at (ln: 15 fl: path_tests/path15.c)

path_tests/path7.c @@@analyzing path_tests/path7.c with testwpa.sh FAIL :NOALIAS check <id:241, id:242> at (ln: 23 fl: path_tests/path7.c) FAIL :NOALIAS check <id:237, id:238> at (ln: 22 fl: path_tests/path7.c)

path_tests/path19.c @@@analyzing path_tests/path19.c with testwpa.sh FAIL :NOALIAS check <id:268, id:236> at (ln: 31 fl: path_tests/path19.c) FAIL :NOALIAS check <id:265, id:232> at (ln: 30 fl: path_tests/path19.c) FAIL :NOALIAS check <id:262, id:236> at (ln: 29 fl: path_tests/path19.c) FAIL :NOALIAS check <id:259, id:232> at (ln: 28 fl: path_tests/path19.c) FAIL :NOALIAS check <id:245, id:240> at (ln: 16 fl: path_tests/path19.c) FAIL :NOALIAS check <id:242, id:240> at (ln: 15 fl: path_tests/path19.c)

path_tests/path12.c @@@analyzing path_tests/path12.c with testwpa.sh FAIL :NOALIAS check <id:243, id:245> at (ln: 15 fl: path_tests/path12.c) FAIL :NOALIAS check <id:239, id:241> at (ln: 14 fl: path_tests/path12.c) [AndersenWPA] Checking NOALIAS path_tests/path2.c @@@analyzing path_tests/path2.c with testwpa.sh FAIL :NOALIAS check <id:249, id:250> at (ln: 28 fl: path_tests/path2.c) FAIL :NOALIAS check <id:246, id:247> at (ln: 27 fl: path_tests/path2.c)

path_tests/path1.c @@@analyzing path_tests/path1.c with testwpa.sh FAIL :NOALIAS check <id:241, id:242> at (ln: 13 fl: path_tests/path1.c) FAIL :NOALIAS check <id:237, id:238> at (ln: 12 fl: path_tests/path1.c)

path_tests/path11.c @@@analyzing path_tests/path11.c with testwpa.sh FAIL :NOALIAS check <id:244, id:245> at (ln: 23 fl: path_tests/path11.c) FAIL :NOALIAS check <id:240, id:241> at (ln: 22 fl: path_tests/path11.c)

path_tests/path14.c @@@analyzing path_tests/path14.c with testwpa.sh FAIL :NOALIAS check <id:259, id:260> at (ln: 23 fl: path_tests/path14.c) FAIL :NOALIAS check <id:255, id:256> at (ln: 22 fl: path_tests/path14.c) FAIL :NOALIAS check <id:251, id:252> at (ln: 21 fl: path_tests/path14.c)

path_tests/path9.c @@@analyzing path_tests/path9.c with testwpa.sh FAIL :NOALIAS check <id:252, id:253> at (ln: 29 fl: path_tests/path9.c) FAIL :NOALIAS check <id:246, id:247> at (ln: 28 fl: path_tests/path9.c)

path_tests/path10.c @@@analyzing path_tests/path10.c with testwpa.sh FAIL :NOALIAS check <id:244, id:234> at (ln: 23 fl: path_tests/path10.c) FAIL :NOALIAS check <id:240, id:241> at (ln: 22 fl: path_tests/path10.c)

yuleisui commented 4 years ago

It seems something wrong with your setting. Have you done the testing using the latest SVF?

bitcalc commented 4 years ago

I pulled the lastest SVF to commit 8c4d8c285a8ac6d16089689051c533a813675d14 which was made on Apr 29. Some failures are gone. The remaining are as follows:

BTW, I specified the following folders:

TestFolders="basic_c_tests\
  fs_tests\
  cs_tests\
  path_tests\
  complex_tests\
  mta\
  mem_leak\
  "
fs_tests/test-su.c
@@@analyzing fs_tests/test-su.c with testwpa.sh
         FAIL :NOALIAS check <id:231, id:233> at (ln: 12 fl: fs_tests/test-su.c)
         FAIL :NOALIAS check <id:226, id:228> at (ln: 11 fl: fs_tests/test-su.c)
--
fs_tests/pcycle1.c
@@@analyzing fs_tests/pcycle1.c with testwpa.sh
         FAIL :NOALIAS check <id:230, id:231> at (ln: 14 fl: fs_tests/pcycle1.c)
--
fs_tests/array_alias_2.c
@@@analyzing fs_tests/array_alias_2.c with testwpa.sh
         FAIL :NOALIAS check <id:281, id:285> at (ln: 32 fl: fs_tests/array_alias_2.c)
         FAIL :NOALIAS check <id:273, id:277> at (ln: 31 fl: fs_tests/array_alias_2.c)
--
fs_tests/global_2.c
@@@analyzing fs_tests/global_2.c with testwpa.sh
         FAIL :NOALIAS check <id:222, id:225> at (ln: 18 fl: fs_tests/global_2.c)
--
fs_tests/strong_update.c
@@@analyzing fs_tests/strong_update.c with testwpa.sh
         FAIL :NOALIAS check <id:236, id:237> at (ln: 14 fl: fs_tests/strong_update.c)
--
fs_tests/global_1.c
@@@analyzing fs_tests/global_1.c with testwpa.sh
         FAIL :NOALIAS check <id:216, id:219> at (ln: 13 fl: fs_tests/global_1.c)
--
fs_tests/struct_2.c
@@@analyzing fs_tests/struct_2.c with testwpa.sh
         FAIL :NOALIAS check <id:230, id:233> at (ln: 22 fl: fs_tests/struct_2.c)
         FAIL :NOALIAS check <id:223, id:226> at (ln: 21 fl: fs_tests/struct_2.c)
--
fs_tests/array_alias_1.c
@@@analyzing fs_tests/array_alias_1.c with testwpa.sh
         FAIL :NOALIAS check <id:228, id:232> at (ln: 25 fl: fs_tests/array_alias_1.c)
         FAIL :NOALIAS check <id:219, id:223> at (ln: 24 fl: fs_tests/array_alias_1.c)
--
fs_tests/struct_1.c
@@@analyzing fs_tests/struct_1.c with testwpa.sh
         FAIL :NOALIAS check <id:230, id:233> at (ln: 22 fl: fs_tests/struct_1.c)
         FAIL :NOALIAS check <id:223, id:226> at (ln: 21 fl: fs_tests/struct_1.c)
--
fs_tests/global_4.c
@@@analyzing fs_tests/global_4.c with testwpa.sh
         FAIL :NOALIAS check <id:218, id:219> at (ln: 12 fl: fs_tests/global_4.c)
--
cs_tests/recur0.c
@@@analyzing cs_tests/recur0.c with testwpa.sh
         FAIL :NOALIAS check <id:221, id:213> at (ln: 11 fl: cs_tests/recur0.c)
--
cs_tests/cs18.c
@@@analyzing cs_tests/cs18.c with testwpa.sh
         FAIL :NOALIAS check <id:243, id:236> at (ln: 21 fl: cs_tests/cs18.c)
         FAIL :NOALIAS check <id:234, id:236> at (ln: 17 fl: cs_tests/cs18.c)
--
cs_tests/recur3.c
@@@analyzing cs_tests/recur3.c with testwpa.sh
         FAIL :NOALIAS check <id:236, id:224> at (ln: 23 fl: cs_tests/recur3.c)
         FAIL :NOALIAS check <id:226, id:228> at (ln: 17 fl: cs_tests/recur3.c)
--
cs_tests/cs17.c
@@@analyzing cs_tests/cs17.c with testwpa.sh
         FAIL :NOALIAS check <id:297, id:299> at (ln: 32 fl: cs_tests/cs17.c)
         FAIL :NOALIAS check <id:292, id:294> at (ln: 31 fl: cs_tests/cs17.c)
         FAIL :NOALIAS check <id:274, id:275> at (ln: 26 fl: cs_tests/cs17.c)
--
cs_tests/cs3.c
@@@analyzing cs_tests/cs3.c with testwpa.sh
         FAIL :NOALIAS check <id:287, id:289> at (ln: 34 fl: cs_tests/cs3.c)
         FAIL :NOALIAS check <id:273, id:275> at (ln: 31 fl: cs_tests/cs3.c)
--
cs_tests/cs4.c
@@@analyzing cs_tests/cs4.c with testwpa.sh
         FAIL :NOALIAS check <id:267, id:269> at (ln: 21 fl: cs_tests/cs4.c)
         FAIL :NOALIAS check <id:262, id:264> at (ln: 20 fl: cs_tests/cs4.c)
--
cs_tests/cs9.c
@@@analyzing cs_tests/cs9.c with testwpa.sh
         FAIL :NOALIAS check <id:282, id:284> at (ln: 25 fl: cs_tests/cs9.c)
         FAIL :NOALIAS check <id:232, id:234> at (ln: 11 fl: cs_tests/cs9.c)
--
cs_tests/cs2.c
@@@analyzing cs_tests/cs2.c with testwpa.sh
         FAIL :NOALIAS check <id:289, id:291> at (ln: 37 fl: cs_tests/cs2.c)
         FAIL :NOALIAS check <id:276, id:278> at (ln: 34 fl: cs_tests/cs2.c)
--
cs_tests/cs20.c
@@@analyzing cs_tests/cs20.c with testwpa.sh
         FAIL :NOALIAS check <id:259, id:232> at (ln: 20 fl: cs_tests/cs20.c)
         FAIL :NOALIAS check <id:234, id:236> at (ln: 12 fl: cs_tests/cs20.c)
--
cs_tests/recur4.c
@@@analyzing cs_tests/recur4.c with testwpa.sh
         FAIL :NOALIAS check <id:239, id:231> at (ln: 27 fl: cs_tests/recur4.c)
         FAIL :NOALIAS check <id:233, id:234> at (ln: 19 fl: cs_tests/recur4.c)
--
cs_tests/recur10.c
@@@analyzing cs_tests/recur10.c with testwpa.sh
         FAIL :NOALIAS check <id:227, id:218> at (ln: 17 fl: cs_tests/recur10.c)
         FAIL :NOALIAS check <id:223, id:225> at (ln: 16 fl: cs_tests/recur10.c)
--
cs_tests/funcpoiner.c
@@@analyzing cs_tests/funcpoiner.c with testwpa.sh
         FAIL :NOALIAS check <id:242, id:243> at (ln: 23 fl: cs_tests/funcpoiner.c)
         FAIL :NOALIAS check <id:232, id:233> at (ln: 19 fl: cs_tests/funcpoiner.c)
--
cs_tests/cs16.c
@@@analyzing cs_tests/cs16.c with testwpa.sh
         FAIL :NOALIAS check <id:243, id:245> at (ln: 23 fl: cs_tests/cs16.c)
         FAIL :NOALIAS check <id:238, id:240> at (ln: 22 fl: cs_tests/cs16.c)
         FAIL :NOALIAS check <id:233, id:235> at (ln: 21 fl: cs_tests/cs16.c)
--
cs_tests/cs11.c
@@@analyzing cs_tests/cs11.c with testwpa.sh
         FAIL :NOALIAS check <id:237, id:238> at (ln: 17 fl: cs_tests/cs11.c)
         FAIL :NOALIAS check <id:233, id:234> at (ln: 16 fl: cs_tests/cs11.c)
--
cs_tests/cs7.c
@@@analyzing cs_tests/cs7.c with testwpa.sh
         FAIL :NOALIAS check <id:265, id:267> at (ln: 27 fl: cs_tests/cs7.c)
         FAIL :NOALIAS check <id:260, id:262> at (ln: 26 fl: cs_tests/cs7.c)
--
cs_tests/cs19.c
@@@analyzing cs_tests/cs19.c with testwpa.sh
         FAIL :NOALIAS check <id:218, id:219> at (ln: 12 fl: cs_tests/cs19.c)
--
cs_tests/cs0.c
@@@analyzing cs_tests/cs0.c with testwpa.sh
         FAIL :NOALIAS check <id:228, id:229> at (ln: 16 fl: cs_tests/cs0.c)
         FAIL :NOALIAS check <id:225, id:226> at (ln: 15 fl: cs_tests/cs0.c)
--
path_tests/path6.c
@@@analyzing path_tests/path6.c with testwpa.sh
         FAIL :NOALIAS check <id:251, id:252> at (ln: 21 fl: path_tests/path6.c)
         FAIL :NOALIAS check <id:247, id:248> at (ln: 20 fl: path_tests/path6.c)
--
path_tests/path17.c
@@@analyzing path_tests/path17.c with testwpa.sh
         FAIL :NOALIAS check <id:255, id:256> at (ln: 26 fl: path_tests/path17.c)
--
path_tests/path3.c
@@@analyzing path_tests/path3.c with testwpa.sh
         FAIL :NOALIAS check <id:243, id:244> at (ln: 24 fl: path_tests/path3.c)
         FAIL :NOALIAS check <id:240, id:241> at (ln: 23 fl: path_tests/path3.c)
--
path_tests/path5.c
@@@analyzing path_tests/path5.c with testwpa.sh
         FAIL :NOALIAS check <id:254, id:255> at (ln: 28 fl: path_tests/path5.c)
         FAIL :NOALIAS check <id:244, id:245> at (ln: 26 fl: path_tests/path5.c)
--
path_tests/path21.c
@@@analyzing path_tests/path21.c with testwpa.sh
         FAIL :NOALIAS check <id:241, id:242> at (ln: 24 fl: path_tests/path21.c)
         FAIL :NOALIAS check <id:238, id:239> at (ln: 23 fl: path_tests/path21.c)
--
path_tests/path18.c
@@@analyzing path_tests/path18.c with testwpa.sh
         FAIL :NOALIAS check <id:244, id:231> at (ln: 16 fl: path_tests/path18.c)
         FAIL :NOALIAS check <id:240, id:242> at (ln: 15 fl: path_tests/path18.c)
         FAIL :NOALIAS check <id:237, id:235> at (ln: 14 fl: path_tests/path18.c)
--
path_tests/path4.c
@@@analyzing path_tests/path4.c with testwpa.sh
         FAIL :NOALIAS check <id:240, id:241> at (ln: 26 fl: path_tests/path4.c)
--
path_tests/path20.c
@@@analyzing path_tests/path20.c with testwpa.sh
         FAIL :MUSTALIAS check <id:224, id:225> at (ln: 18 fl: path_tests/path20.c)
--
path_tests/path8.c
@@@analyzing path_tests/path8.c with testwpa.sh
         FAIL :NOALIAS check <id:264, id:265> at (ln: 30 fl: path_tests/path8.c)
         FAIL :NOALIAS check <id:260, id:261> at (ln: 29 fl: path_tests/path8.c)
--
path_tests/path13.c
@@@analyzing path_tests/path13.c with testwpa.sh
         FAIL :NOALIAS check <id:253, id:254> at (ln: 26 fl: path_tests/path13.c)
         FAIL :NOALIAS check <id:249, id:250> at (ln: 25 fl: path_tests/path13.c)
--
path_tests/path16.c
@@@analyzing path_tests/path16.c with testwpa.sh
         FAIL :NOALIAS check <id:249, id:250> at (ln: 23 fl: path_tests/path16.c)
         FAIL :NOALIAS check <id:245, id:246> at (ln: 22 fl: path_tests/path16.c)
--
path_tests/path15.c
@@@analyzing path_tests/path15.c with testwpa.sh
         FAIL :NOALIAS check <id:224, id:225> at (ln: 15 fl: path_tests/path15.c)
--
path_tests/path7.c
@@@analyzing path_tests/path7.c with testwpa.sh
         FAIL :NOALIAS check <id:241, id:242> at (ln: 23 fl: path_tests/path7.c)
         FAIL :NOALIAS check <id:237, id:238> at (ln: 22 fl: path_tests/path7.c)
--
path_tests/path19.c
@@@analyzing path_tests/path19.c with testwpa.sh
         FAIL :NOALIAS check <id:268, id:236> at (ln: 31 fl: path_tests/path19.c)
         FAIL :NOALIAS check <id:265, id:232> at (ln: 30 fl: path_tests/path19.c)
         FAIL :NOALIAS check <id:262, id:236> at (ln: 29 fl: path_tests/path19.c)
         FAIL :NOALIAS check <id:259, id:232> at (ln: 28 fl: path_tests/path19.c)
         FAIL :NOALIAS check <id:245, id:240> at (ln: 16 fl: path_tests/path19.c)
         FAIL :NOALIAS check <id:242, id:240> at (ln: 15 fl: path_tests/path19.c)
--
path_tests/path12.c
@@@analyzing path_tests/path12.c with testwpa.sh
         FAIL :NOALIAS check <id:243, id:245> at (ln: 15 fl: path_tests/path12.c)
         FAIL :NOALIAS check <id:239, id:241> at (ln: 14 fl: path_tests/path12.c)
--
path_tests/path2.c
@@@analyzing path_tests/path2.c with testwpa.sh
         FAIL :NOALIAS check <id:249, id:250> at (ln: 28 fl: path_tests/path2.c)
         FAIL :NOALIAS check <id:246, id:247> at (ln: 27 fl: path_tests/path2.c)
--
path_tests/path1.c
@@@analyzing path_tests/path1.c with testwpa.sh
         FAIL :NOALIAS check <id:241, id:242> at (ln: 13 fl: path_tests/path1.c)
         FAIL :NOALIAS check <id:237, id:238> at (ln: 12 fl: path_tests/path1.c)
--
path_tests/path11.c
@@@analyzing path_tests/path11.c with testwpa.sh
         FAIL :NOALIAS check <id:244, id:245> at (ln: 23 fl: path_tests/path11.c)
         FAIL :NOALIAS check <id:240, id:241> at (ln: 22 fl: path_tests/path11.c)
--
path_tests/path14.c
@@@analyzing path_tests/path14.c with testwpa.sh
         FAIL :NOALIAS check <id:259, id:260> at (ln: 23 fl: path_tests/path14.c)
         FAIL :NOALIAS check <id:255, id:256> at (ln: 22 fl: path_tests/path14.c)
         FAIL :NOALIAS check <id:251, id:252> at (ln: 21 fl: path_tests/path14.c)
--
path_tests/path9.c
@@@analyzing path_tests/path9.c with testwpa.sh
         FAIL :NOALIAS check <id:252, id:253> at (ln: 29 fl: path_tests/path9.c)
         FAIL :NOALIAS check <id:246, id:247> at (ln: 28 fl: path_tests/path9.c)
--
path_tests/path10.c
@@@analyzing path_tests/path10.c with testwpa.sh
         FAIL :NOALIAS check <id:244, id:234> at (ln: 23 fl: path_tests/path10.c)
         FAIL :NOALIAS check <id:240, id:241> at (ln: 22 fl: path_tests/path10.c)
yuleisui commented 4 years ago

There should be no failures at all. It works all good on my machine. Could you test each folder at one time. Try basic c tests first and then change scripts testwpa.sh and try fs tests.

yuleisui commented 4 years ago

You can not use one analysis to test all types of benchmarks. For fs tests, the correct option is -fspta in testwpa.sh. For context and path sensitive tests, please refer to the options in SUPA https://github.com/SVF-tools/SUPA.

bitcalc commented 4 years ago

I see. When I test the fs_tests folder with -fspta option, there was not a failure. Strangely, the link https://github.com/SVF-tools/SUPA doesn't list that option. Where can I find which option I should use for each test folder?

yuleisui commented 4 years ago

For cs test, you can use the option in 3.2 on SUPA's page (change it accordingly in testdvf.sh). For path test, it is not supported in the public SVF yet.

yuleisui commented 4 years ago

For men leaks, please use testsaber.sh

bitcalc commented 4 years ago

Thank you very much for helping me out.

I have a few suggestions after this bumping experience of running tests. They should help me and future newcomers to speed up.

  1. There should be a single command to run all tests with proper options. It isn't hard to imagine that some changes in one analysis might affect the result of another. It's troublesome to manually change the script and run all tests each time. This also helps automatic nightly run of all tests.

  2. Add another column in the bottom table on the README.md page listing the command and options required to test a folder.

  3. Create a stable branch where all tests should pass. This will help me and other newcomers know where to look for the correct baseline. This might require you to synchronize the PTABen and SVF repositories with tags, or just moving current test suites into the SVF repository and leaving the PTABen for really big projects.

Cheers!

yuleisui commented 4 years ago

Yes, thanks for the suggestions. I am not good at writing scripts. I wrote the automated testing scripts few years ago. It's time to integrate them in the nightly build or Travis. Would you happy to help with the first and second suggestions? Thanks!

bitcalc commented 4 years ago

I can help with 2. But I'm not good at writing bash scripts either. I can read and change them, though.

yuleisui commented 4 years ago

Thanks, that will be good!

bitcalc commented 4 years ago

For mem_leak, testsaber.sh looks for a program called instr under the build/bin directory. I don't have it. Was it renamed, or should I build that program somehow?

yuleisui commented 4 years ago

Instr is not used anymore. Please simply comment out it and I will upload a correct one tomorrow.

yuleisui commented 4 years ago

Just comment all instructions for 'Saber instrumentation', it will work.

bitcalc commented 4 years ago

Yup, it works. Thank you. I'll make a pull request for the update.