RFE - ability to add constraints to tests

[apnadkarni]

The tcltest package that is part of the Tcl core has the -constraints command line option that permits specification of constraints. In addition, constraints can be attached to tests. If constraints are attached to a test, it is only run if the corresponding constraint is enabled via the command line or programmatically. For example, there can be constraints that specify that tests only run on Windows, or only on multiprocessor systems or only if certain software is installed.

A similar capability for Caius would be useful.

[tobijk]

What would be the best way to attach constraints to tests in your view? This is a cool feature, but I'm afraid it would require custom commands to construct test methods...

But it could definitely be a testplan feature to allow filtering based on constraints defined in XML attributes.

[apnadkarni]

The way constraints are used in the Tcl core test suite (and the way I use them in my own software) is to prevent tests from running in environments that do not support them.

One example is the registry command which is not relevant on platforms other than Windows. For this case, the testplan filtering based on XML attributes would work since the entire module is constrained.

Another example is when some specific options are only supported on some platforms. For example, the file attributes command has options specific to each platform. In this case, testplan attributes would not suffice.

But I see why this would not be easy to implement on a per-test method basis.

[tobijk]

Here is an idea for constraints in tests. We add a skip method to Testing::TestObject. Skip is called like this:

skip ?-constraints <constraint1>[,<constraint2>,..]?

skip checks if all constraints are satisfied, if not, it raises a special TestSkipped exception, which is caught in run and causes the test to be ignored.

Ok, the naming is maybe not so great, but you get the idea.

[apnadkarni]

So skip would be called from the top of the test method? That would work fine I think.

Ideally, the constraints should be settable/unsettable with a method call on TestObject and also passable from the command line.

I agree skip may not be an ideal name, almost the opposite of what it does. Could we just call the method constraints and pass a list of the constraint values ?

constraints win32 pointer64

would run the test only on 64-bit Windows platforms.

The above like tcltest, requires all constraints to be satisfied. To go Tcl one better, and make it more flexible perhaps an expression could be passed

constraints {$win32 && $pointer64 && ! $running_as_administrator}

Visually not as clean but more flexible. The implementation itself might be something like (using TclOO syntax as not familiar with ITcl)

method add_constraints args {
   variable constraints_dictionary
   foreach constraint $args {dict set constraints_dictionary $arg 1}
}
method constraints args {
    variable constraints_dictionary
    dict with $constraints_dictionary {
        if {[expr $args]} return
        error "Skip test" "" TestSkipped
    }
}

[tobijk]

Ok, let me think about it a bit more.

Somewhat unrelated, I found that pointer64 bit interesting. But according to this listing [http://wiki.tcl.tk/1649]() it's not a reliable way to determine OS' bits.

[tobijk]

I tried to implement constraints in 1702b5e9, but there are some problems:

• setup and teardown are still going to be run
• logging cannot be suppressed
• marking tests as skipped possibly doesn't work with Junit or other log formats

Doing this properly requires a custom command for defining test methods.

[tobijk]

I decided to go for an #ifdef kind of approach. I understand that this is not 1:1 what tcltest does, but I think it serves the purpose. This is fixed in master.

[apnadkarni]

This is quite reasonable. I don't think it needs to exactly match tcltest. Thanks

/Ashok