nus-cs2103-AY2223S2 / forum

12 stars 0 forks source link

[Practice Paper] Path coverage #423

Closed BoAi01 closed 1 year ago

BoAi01 commented 1 year ago

A follow-up for the below question: what is the number of test cases needed to achieve 100% path coverage? Is it that we need to exhaust all possible numbers of iterations for the for loop (i.e. infinitely many cases or just all possible values for int)?

Screenshot 2023-04-25 at 4 23 24 PM
hingen commented 1 year ago

I think one way to think about path coverage is to list out all the possible sequences of lines of code that can be executed. If you can write test cases to cover all these sequences, you've achieved 100% path coverage. For this question it would be: 234 (s is null) 2568 (s is empty string) 256768 (s is length 1) 25676768 (s is length 2) 2567676768 (s is length 3) and so on ...

Achieving 100% path coverage would involve testing all possible string lengths which (if I'm not wrong) is up to Integer.MAX_VALUE since String.length() returns an int.

joellow88 commented 1 year ago

I think one way to think about path coverage is to list out all the possible sequences of lines of code that can be executed. If you can write test cases to cover all these sequences, you've achieved 100% path coverage. For this question it would be: 234 (s is null) 2568 (s is empty string) 256768 (s is length 1) 25676768 (s is length 2) 2567676768 (s is length 3) and so on ...

Achieving 100% path coverage would involve testing all possible string lengths which (if I'm not wrong) is up to Integer.MAX_VALUE since String.length() returns an int.

But doesn't this definition makes path coverage a very unhelpful metric? To achieve 100% path coverage, we essentially need 2 billion test cases. Suppose we write 5 test cases for this function, path coverage would be ~1E-7%, which doesn't really help us design better test cases.

hingen commented 1 year ago

But doesn't this definition makes path coverage a very unhelpful metric?

I think when designing test cases, it's likely good to look at multiple different coverage metrics. Getting 100% path coverage is not always the most efficient approach (definitely not in this case). However, in situations where it is achieved, it is a pretty good sign that your test cases are very comprehensive.

Perhaps looking at multiple coverage metrics can help give one an idea of how much likely a bug is to occur in a particular area of the code. For example, a method with 100% path coverage is likely less bug prone than a method with a lower path coverage. Can possibly help in deciding where to focus your attention for exploratory testing.

damithc commented 1 year ago

To add to what @hingen said, in this specific example, the infeasibility of achieving 100% path coverage also makes us wonder if there is a limit in the size of s and if we should make the code more defensive to match that limit e.g., add assert s.length() < 100; while also making the 100% path coverage more feasible.

EvitanRelta commented 1 year ago

What exactly is a "path" in "path coverage" ?

Does the below code also have zillions of paths like the Practice Paper question?
As printString("a") prints a different string from printString("b") ?

static void printString(String s) {
    print(s);
}


And does the below method need Integer.MAX_VALUE - Integer.MIN_VALUE number of test cases to achieve 100% path coverage?
Or does it need Integer.MAX_VALUE number of test cases only, since non-positive number all iterates 0 times? Or does it need 1 test case, since the loop does nothing?

static void loopNothing(int numOfLoops) {
    for (int i = 0; i < numOfLoops; i++) {}
}
damithc commented 1 year ago

What exactly is a "path" in "path coverage" ?

@EvitanRelta A path is a unique sequence of statements (i.e., nodes) that the execution can take.

Suppose we have the code:

void foo(String p) {
    if (x) {  // 1
        print("hi");  //2
    }
    print(p)  //3
}

Possible paths for the above code are: 1 3 1 2 3

The fact that p can vary every time doesn't affect the number of paths.

BoAi01 commented 1 year ago

A path is a unique sequence of statements (i.e., nodes) that the execution can take.

Great! This sounds reasonable.

And does the below method need Integer.MAX_VALUE - Integer.MIN_VALUE number of test cases to achieve 100% path coverage?

Frankly, I think so, but we don't need to aim for 100% path coverage though. Sampling representative values from different partitions should be good enough. Exhaustive testing is not practical -- the possible paths in a large program are combinatorial... not possible to test all of them.