How to find the input that generates an assert failure in the test tests/file/large_int.bas

[CharlesDDNoble]

Hello FBC devs,

I'm working on a research project that involves analyzing bugs and the specific input that generates them. I'm evaluating a previous version of FBC that is included in the ManyBugs repository (info for this can be found here, the specific version can be found in the scenario: fbc-bug-5251-5252). The bug in question has to do with the added functionality of octal support using '&'. There are two test files that I am particularly interested: tests/string/val.bas and tests/file/large_int.bas.

My question is this: In the large_int.bas test file, how can I determine which input causes one of the included assertions to fail?

When I run the executable created by compiling large_int.bas, it lists the assertions that fail like so,

1. file/large_int.bas:162 - CU_ASSERT_DOUBLE_EQUAL( check(i), n, check(i) \ 10000000 )

I would like to determine which specific input i, like "&2000000", leads to an assert failure. I have included sections of the code that may be pertinent to answering this.

This is the section of code that uses assertions:

#macro TYPE_TEST( t )
    open "./file/large_int.csv" for input as #1
        for i as integer = 0 to COUNT-1
        dim n as t
        input #1, n
 #if t = single 
        CU_ASSERT_DOUBLE_EQUAL( check(i), n, check(i) \ 10000000 )
 #elseif t = double
        CU_ASSERT_DOUBLE_EQUAL( check(i), n, check(i) \ 1000000000000000ll )
 #else
        if( clngint( cast( t, check(i) ) ) = check(i) ) then
            CU_ASSERT_EQUAL( check(i), clngint(n) )
        end if
 #endif
    next i
    close #1
#endmacro

This section uses the macro:

TYPE_TEST(     byte )
TYPE_TEST(    ubyte )
TYPE_TEST(    short )
TYPE_TEST(   ushort )
TYPE_TEST(  integer )
TYPE_TEST( uinteger )
TYPE_TEST(     long )
TYPE_TEST(    ulong )
TYPE_TEST(  longint )
TYPE_TEST( ulongint )
TYPE_TEST(   single )
TYPE_TEST(   double )

This section generates the input file that is being opened and tested:

sub generate_csv () constructor

        dim n as longint

        open "./file/large_int.csv" for output as #1

            ''decimal
            n = 1
            for i as integer = 1 to DEC_COUNT
                write #1, n, -n, n * 10 - 1, 1 - n * 10
                n *= 10
            next i
            write #1, n, -n

            print #1,  "9223372036854775807" '' 1ull shl 63 - 1
            print #1, "-9223372036854775808" '' -1ll shl 63

            ''hex
            n = 1
            for i as integer = 1 to HEX_COUNT
                print #1, "&h" & hex(n) & ", &h" & hex(n shl 3 - 1)
                print #1, "&h" & hex(n shl 3) & ", &h" & hex(n shl 4 - 1)
                n shl= 4
            next i

            ''oct
            n = 1
            for i as integer = 1 to OCT_COUNT
                print #1, "&o" & oct(n) & ", &o" & oct(n shl 3 - 1)
                n shl= 3
            next i

            ''bin
            n = 1
            for i as integer = 1 to BIN_COUNT
                print #1, "&b" & bin(n) & ", &b" & bin(n shl 1 - 1)
                n shl= 1
            next i

            ''misc
            print #1, "&b11, &B11, &o77, &O77, &77, &hff, &HFF"  '' 3, 3, 63, 63, 63, 255, 255
            print #1, "&hf, &hF, &Hf, &HF"                  '' 15, 15, 15, 15

            print #1, "1.23d+2, 1.2D+1, 3.4e+1, 3.45E+2"    '' 123, 12, 34, 345

            print #1, "   1d1,    2D2,    3e3,    4E4"      '' 10, 200, 3000, 40000
            print #1, "  1.d1,   2.D2,   3.e3,   4.E4"      '' 10, 200, 3000, 40000
            print #1, " &h1d1,  &h2D2,  &h3e3,  &h4E4"      '' 465, 722, 995, 1252
            print #1, "&h1.d1, &h2.D2, &h3.e3, &h4.E4"      '' 1,  2,   3,    4

            print #1, "&b12, &o78, &hfg, &HFG"              '' 1, 7, 15, 15

        close #1

    end sub

Thanks for your time!

[rversteegen]

Sorry for the slow reply; I've been really busy.

Oh, that's a cool project! Did fbc end up in that dataset, with just 8 other programs, just because it meets all the criteria? It seems strange though, because fbc (the compiler) is written in FreeBasic, not C, so its inclusion appears invalid. Only the runtime libraries are written in C. FB is exceedingly closely related to C though; it is basically a subset of C++98 with BASIC syntax (optionally QuickBASIC-compatible) and standard library. The bug you are investigating may be in either FB or C code, I don't know. Also, it's not accurate to describe FB as a "legacy language compiler" since it's its own, new language rather than a QuickBASIC revival project :)

You're going to have to edit file/large_int.bas, by adding this line before the failing CU_ASSERT_DOUBLE_EQUAL: if (abs(cdbl(check(i))-cdbl(n)) <= abs(cdbl(check(i) \ 10000000))) then print "failure with " & check(i) & " " & n This just copies the test that CU_ASSERT_DOUBLE_EQUAL expands to.

[jayrm]

@CharlesDDNoble, yes, interesting project. We consider unit testing an important part of our development, so very cool to see fbc used for input and study in your research.

The test-suite's CUASSERT* macros generate a generic message that is suitable for most cases where we want to know the expression that failed at a specific file/line number. But as you have discovered, since the context of the assertion is dependent on an input file, the full context is not reported.

The older versions of fbc use CUnit for the testing framework. Around December 2017, we re-implemented it as fbcunit written in freebasic instead of C, and using a simplified API, However, most of the same macro names are still used across both.

CUnit (fbc 1.05.0 and earlier)

fbc's CU_ASSERT_DOUBLE_EQUAL is defined in inc/CUnit/CUnit.bi

#define CU_ASSERT_DOUBLE_EQUAL(actual, expected, granularity) CU_assertImplementation(-(fabs(cdbl(actual) - (expected)) <= fabs(cdbl(granularity))), __LINE__, "CU_ASSERT_DOUBLE_EQUAL(" #actual "," #expected "," #granularity ")", __FILE__, "", CU_FALSE)

I think the most direct way to get more specific results but still in standard format is to define a new assertion macro, for example, CU_ASSERT_DOUBLE_EQUAL_MSG to include an extra message to report the specific values:

#define CU_ASSERT_DOUBLE_EQUAL_MSG(actual, expected, granularity, msg) CU_assertImplementation(-(fabs(cdbl(actual) - (expected)) <= fabs(cdbl(granularity))), __LINE__, "CU_ASSERT_DOUBLE_EQUAL(" #actual "," #expected "," #granularity ") " & msg, __FILE__, "", CU_FALSE)

Then use the new macro, for more specific results:

CU_ASSERT_DOUBLE_EQUAL_MSG( check(i), n, check(i) \ 10000000, "check(" & i & ")=" & check(i) & ", n=" & n )

fbcunit (fbc 1.06.0 and later)

This might make a nice addition, or some variation of it, to current fbcunit test-suite API. This framework is located in https://github.com/freebasic/fbc/tree/master/tests/fbcunit

Solution would be similar:

CU_ASSERT_DOUBLE_EQUAL is one of many #defines that wraps the test-suite's fbcu.CU_ASSERT_ generic assertion function found in tests/fbcunit/inc/fbcunit.bi:

#define CU_ASSERT_DOUBLE_EQUAL( a, e, g ) fbcu.CU_ASSERT_( (abs(cdbl(a)-cdbl(e)) <= abs(cdbl(g))), __FILE__, __LINE__, __FUNCTION__, "CU_ASSERT_DOUBLE_EQUAL(" #a "," #e "," #g ")" )

Where fbcu.CUASSERT is declared as:

namespace fbcu
    declare sub CU_ASSERT_ _
        ( _
            byval value as boolean, _
            byval fil as zstring ptr, _
            byval lin as long, _
            byval fun as zstring ptr, _
            byval msg as zstring ptr _
        )
end namespace

value := expression that evaluates to true (pass) or false (fail)
fil   := filename (of the test source file)
lin   := line number (of the test source file)
fun   := function name (of the test source file)
msg   := custom message

[CharlesDDNoble]

@rversteegen @jayrm Thanks a lot! I really appreciate the quick feedback!

@rversteegen I am not actually a member of the team that published the paper and created the repository, but I believe FBC was included because it has suffient C source code and some defects in the version history include only changes to C source code (Python and PHP are also projects included for this reason). Many Auto Program Repair (APR) systems are made specifically for C and the purpose of the ManyBugs repository is to provide a system to test APR's.

@jayrm Thanks for the info on the the FBC testing framework, this will come in handy when I am looking into the newer versions.

I ultimately used the method suggested by @rversteegen of just printing the information before the actual assert. I added the code mentioned before the assert (changing the '<=' to '>' to report failures). This is what I used:

if (abs(cdbl(check(i))-cdbl(n)) > abs(cdbl(check(i) \ 10000000))) then print "failure with " & check(i) & " (" & i & ") " & n

Thanks again!

[rversteegen]

I'd forgotten about the fbcunit rewrite, so that was helpful for me too!