neopragma / cobol-unit-test

Unit testing framework and sample code for batch Cobol programs.
GNU General Public License v3.0
117 stars 36 forks source link

Labels for Pass/Fail Notifications are Inconsistent #73

Open d215steinberg opened 9 years ago

d215steinberg commented 9 years ago

1) Pass/Fail Notifications for EXPECT assertions are labeled with test name (duplicating test name when multiple EXPECTs under test) 2) Pass/Fail Notifications for VERIFY assertions are labeled with generated phrases 3) There is no indication of the hierarchical relationship of test cases and assertions

EXAMPLE (based on FILEDEMT):

TESTCASE 'IT WORKS FOR THE SIMPLE STUFF'
       EXPECT '1' TO BE '1'
       EXPECT '2' TO BE '2' 

 TESTCASE 'IT MOCKS ERROR ON FILE OPEN INPUT' 
       MOCK
           FILE INPUT-FILE 
           ON OPEN STATUS '87'
       END-MOCK    
       PERFORM 0100-OPEN-INPUT
       EXPECT WS-INPUT-FILE-STATUS TO BE '87'
       EXPECT '1' TO BE '2'
       VERIFY FILE INPUT-FILE READ NEVER HAPPENED
       VERIFY FILE INPUT-FILE OPEN HAPPENED 3 TIMES

Output:

PASS:   4. IT WORKS FOR THE SIMPLE STUFF                                                   
PASS:   5. IT WORKS FOR THE SIMPLE STUFF                                                   
PASS:   6. IT MOCKS ERROR ON FILE OPEN INPUT                                               
**** FAIL:   7. IT MOCKS ERROR ON FILE OPEN INPUT                                               
    EXPECTED <2               >, WAS <1               > 
PASS:   8. VERIFY     0 ACCESSES
**** FAIL:   9. VERIFY ACCESSES TO OPEN ON INPUT-FILE | EXPECTED  3 ACCESSES, WAS 1                           

Expect to see something more like the following:

PASS:  4. IT WORKS FOR THE SIMPLE STUFF
**** FAIL:   5. IT MOCKS ERROR ON FILE OPEN INPUT
    EXPECTED <2               >, WAS <1               > 
    VERIFY ACCESSES TO OPEN ON INPUT-FILE | EXPECTED  3 ACCESSES, WAS 1        

When all assertions of a test case pass, we should report only that the test case has passed. When any assertions of a test case fail, we should report the test case failure along with the details of each failing assertion.

neopragma commented 9 years ago

DN. I'm calling this a bug because of the inaccurate verify count shown in the example above ("output").

It's partly an an enhancement to improve the messaging (testcase text is displayed for every assertion under the given testcase). Something similar to rspec 'describe' and 'it' might be preferable.