Uberi / Yunit

Super simple testing framework for AutoHotkey.
GNU Affero General Public License v3.0
53 stars 21 forks source link

Exit Code #2

Open Uberi opened 11 years ago

Uberi commented 11 years ago

The test structure at the moment precludes the usage of exit codes. This is useful for things like Git hooks and the like.

Having exit codes means something else as well: ExitApp, Code must be used to complete the script. That means that there needs to be a post-completion method for when all the test suites given to Yunit have completed, maybe Yunit.Complete(), to run the exit routine. Thought about using OnExit, but the tests themselves might require it.

Yunit.Test(Suites)
Yunit.Complete() ;exits if not using GUI module, for instance, otherwise sets up the exit code in the GuiClose routine.
Return ;this line will run only if using the GUI module or other event driven one.
infogulch commented 11 years ago

Obviously there should be a way to test different possible exit codes in one test suite, as even one small program could have many. This means that it will be impossible to call code that uses ExitApp directly from the test suite, since you cannot exit more than once.

The point of having all these Begin() and End() and categories and subclasses etc is to remove as many side-effects from tests as possible. Unfortunately, ExitApp is a side effect that cannot be recovered from so we can't really support it directly in the tests themselfs.

Perhaps we can make an easy way to run a program itself as a test (say, running a script dynamically), or set up some conventions for dealing with these kinds of unrecoverable side effects.

I think you're mixing up the subject of the test with the test itself. In the case of exit codes the test shouldn't use ExitApp, Code, instead, test should run a program and make sure that the exit code is correct when it's done.

infogulch commented 8 years ago

Uberi, rereading your issue again after aviaryan mentioned it made me realize I misunderstood it at first, almost 4 (_four_!) years ago.

I originally thought that you were talking about giving a test suite the capability to test it's own use of exit codes, which seem to be rather problematic. (On second thought, perhaps an OnExit function could be used for this purpose.)

Anyways, as far as a new response to your original issue: I agree in general, but since the testing context is not saved after the call to Test and the expected use case is to call on the object directly after creating it, perhaps the Test method should return an object that has the Complete() object on it. As in Yunit.Test(suite1, suite2).Complete().

Edit: on second thought, what if the Test() method just returned the number of failed tests? Then you can do what you want with that including ExitApp Yunit.Test(suite)

Uberi commented 8 years ago

Has it really been that long? Feels like just a little while ago :)

I admit, I haven't looked at a lot of this stuff for a while now, but a lot of GUIs still seem to use OnExit for close button handling, so perhaps overriding that isn't a good idea. The ExitApp syntax is definitely very slick, though. Maybe as an option?