Return test results in Yunit.Test

[aviaryan]

It would be better if Yunit.Test returns test results (instance.results variable in code) instead of void. That way one can programatically evaluate the results and do stuff. One possible use it to check the results and exit script with 1 in case a test fails, something which has been already discussed in #2 .

[infogulch]

Maybe, but isn't this the purpose of output Modules? Perhaps you could make a small module that just stores the results into a variable.

The main reasons it was designed to use output modules are:

1. The output is live. Test suites can take a long time and you may not want to wait for a full suite to complete before seeing any output.
2. It's easy to create your own module and simply plug it in, and use any number of output modules at once.

In this case (saving the results to a variable) you don't really care about point 1, but point 2 still stands.

[aviaryan]

1. It's easy to create your own module and simply plug it in, and use any number of output modules at once.

I guess you are right. That is a way to do it and it will be more convenient for my needs but since there is only one format in which you are storing the test results and it is irrespective of the modules used, isn't it plausible to return the results in the method (Test). That way if someone wants, they can use the test results, they won't have to create a module for that.

results := tester.Test(suite1, suite2) ; doesn't it look good

I also have a feeling that returning the associative array instance.results as results will look ugly as it is not properly designed/formatted for end-user use. So you may just return a boolean True/False value in tester.Test().

As you said, that test suites can be long so there maybe some users (or automated systems) who want to exit the tests immediately in case one of the test suite fails.

result = tester.Test(suite1)
if (!result){
    exitapp, 1
}
result := tester.Test(suite2)
...

That should be a fairly good use case. What are your thoughts ?

[infogulch]

If you want an early exit mechanism, you could make a ... drum roll ... module! (I'm sure a "There's a module for that." joke is hidden somewhere here.)

class EarlyExit ;untested
{
    Update(Category, Test, Result)
    {
        if IsObject(Result)
        {
            ExitApp 1
        }
    }
}

Then your test line would be: Yunit.Use(YunitStdout, EarlyExit).Test(suite1, suite2) and it would exit on the first failing test.

Having Test() return a simple value (like the number of failed tests) sounds like a good idea anyways. This is being discussed in #2.

Perhaps some of these convenience modules could be added to the library. Another module idea I can think of would wrap any other module and only pass failed tests on to the wrapped module. This would make those failing tests easier to find when using automated systems (e.g. git bisect).

[aviaryan]

If you want an early exit mechanism, you could make a ... drum roll ... module!

I thought modules were more like "Output" modules and shouldn't use a strong command like ExitApp. BTW, my real use is to know if some test failed after executing all the tests and then ExitApp accordingly. To use a module to do this, either I will have to use some global variable or a module class (static) variable. The later is obviously the better option but using a module for such a basic Test Framework feature is too much.

I like the returning no of tests failed in Test() idea you proposed in #2. Hope that gets implemented.

BTW, by automated systems I meant automated testing (CI). See https://github.com/aviaryan/Ahk-CI-Example/

[infogulch]

I thought modules were more like "Output" modules and shouldn't use a strong command like ExitApp.

I suppose. (In fact I think it would be a good idea to rename Yunit.Modules to OutputModules to make that more clear.)

The only thing I could see that could be a problem is that Update is called before the test suite's End method, which could result in things that aren't cleaned up. But this could be fixed very easily by just moving the call down a few lines.

The only other thing is the suite destructor, but iirc the __Delete method should be called for the test suite when the program exits.

Edit: I like your CI library. I'll definitely use it on my next project.