Failing Test Suite

[MartinNowack]

I opened this issues to streamline discussions here to a single place:

@yuleisui I had a look at the failing test suite:

The issue is that: 6fcf6cbba9c071b9cea541dbac33877223b2dc77 removed all the *.bc files. Therefore, the CI won't find any test cases.

I guess regenerating them and adding them again would be the easiest solution. Still, it would be great if we could transition the test-suite from pre-generated bc files to compiling the files as part of the test run execution. This would also allow you to test multiple LLVM versions in parallel.

I guess it's a bit harder for the more complex ones (crux-bc) but for the smaller it should be fine.

What are your thoughts on that?

[MartinNowack]

Just came to my mind. Did you ever consider lit from the LLVM framework (https://llvm.org/docs/CommandGuide/lit.html)? Running/Writing tests is quite expressive in combination with FileCheck: https://www.llvm.org/docs/CommandGuide/FileCheck.html#tutorial.

[yuleisui]

Can we auto regenerate all the bcs as before? Why they disappear?

[yuleisui]

We have generated_bc.sh. It should auto generate bcs every time when running CI?

[yuleisui]

It seems the compilation problem. https://github.com/SVF-tools/Test-Suite/actions/runs/3591672251/jobs/6046483559#step:5:322

[MartinNowack]

It looks like it's caused by rolling out the new github action builders. Runs are currently building with Ubuntu 22.04 - couple of commits ago it was Ubuntu 20.04.

[yuleisui]

I see. Can we change the action back to 20.04?

[yuleisui]

Just updated back to 20.04. The bcs are generated this time. Let us see whether SVF’s CI can pass or not..

[yuleisui]

Unfortunately, ctest failed. This error is strange. Could you take a look?

https://github.com/SVF-tools/SVF/actions/runs/3591236314/jobs/6054960506#step:9:93

[yuleisui]

It looks to me because of the generated diff_test_osx executable is incorrect.

[MartinNowack]

Similar issue - the software is built for 12 but run on 11 - although it was requested like that. The following might help - let's see. https://github.com/SVF-tools/SVF/pull/939

[yuleisui]

I have changed it to osx-11 here to be consistent with the upstream svf (osx-11 looks unavailable for GitHub action)

https://github.com/SVF-tools/Test-Suite/commit/0e1b883f9c9592841429f29fe594142c3867c450

The error still exists https://github.com/SVF-tools/SVF/actions/runs/3591236314/jobs/6055710540#step:9:1779

[MartinNowack]

We have generated_bc.sh. It should auto generate bcs every time when running CI?

It would be generated for each test case run - by each test case itself.

Something like the following lines could be added to an example test case:

// RUN: %clang %s -g -o %t.bc
// RUN: %wpa -ander -vfspta -dump-vfg 2> %t.log
// RUN: %FileCheck %s --input-file=%t.log 
/*
 * Alias due to function pointer resolution.
 * Author: Sen Ye
 * Date: 06/09/2013
 */
#include "aliascheck.h"

// CHECK: THIS STRING MUST BE PART OF THE OUTPUT
void f(int *m, int *n)
{
    MAYALIAS(m, n);
}

typedef void (*fp)(int*m, int*n);

int main()
{
    int a,b,c,d;
    int *pa, *pb;
    fp p;
    pa = &a, pb = &b;
    f(pa, pb);
    p = f;
    pb = &a;
    (*p)(pa, pb); 
    return 0;
}

This would compile this source file in the first step. Execute it with the provided arguments of the second line and checks the output from the third line. Of course, this could be mixed and matched like you want.

It allows a much more fine-grain control of your test execution and is test specific. You could also add multiple lines like:

// RUN: %clang %s -g -o %t.bc
// RUN: %wpa -foo 2> %t.log.foo
// RUN: %wpa -bar 2> %t.log.bar
// RUN: %wpa -baz 2> %t.log.baz

// RUN: %FileCheck %s --input-file=%t.log.foo
// RUN: %FileCheck %s --input-file=%t.log.bar
// RUN: %FileCheck %s --input-file=%t.log.baz

[yuleisui]

We have generated_bc.sh. It should auto generate bcs every time when running CI?

It would be generated for each test case run - by each test case itself.

Something like the following lines could be added to an example test case:

// RUN: %clang %s -g -o %t.bc
// RUN: %wpa -ander -vfspta -dump-vfg 2> %t.log
// RUN: %FileCheck %s --input-file=%t.log 
/*
 * Alias due to function pointer resolution.
 * Author: Sen Ye
 * Date: 06/09/2013
 */
#include "aliascheck.h"

// CHECK: THIS STRING MUST BE PART OF THE OUTPUT
void f(int *m, int *n)
{
  MAYALIAS(m, n);
}

typedef void (*fp)(int*m, int*n);

int main()
{
  int a,b,c,d;
  int *pa, *pb;
    fp p;
  pa = &a, pb = &b;
  f(pa, pb);
    p = f;
  pb = &a;
    (*p)(pa, pb); 
  return 0;
}

This would compile this source file in the first step. Execute it with the provided arguments of the second line and checks the output from the third line. Of course, this could be mixed and matched like you want.

It allows a much more fine-grain control of your test execution and is test specific. You could also add multiple lines like:

// RUN: %clang %s -g -o %t.bc
// RUN: %wpa -foo 2> %t.log.foo
// RUN: %wpa -bar 2> %t.log.bar
// RUN: %wpa -baz 2> %t.log.baz

// RUN: %FileCheck %s --input-file=%t.log.foo
// RUN: %FileCheck %s --input-file=%t.log.bar
// RUN: %FileCheck %s --input-file=%t.log.baz

Sounds good, but it needs quite a lot of efforts…

The current SVF CI seems working now after changing the test suite one to osx-11

https://github.com/SVF-tools/SVF/actions/runs/3591236314/jobs/6056392014