Allow any Java Object as VarValue

[tkruse]

For my library usage of Tcases, it becomes a liability to have to serialize values to Strings, just to pass to Tcases and extract later from TestCases. The representation of values as Strings in Tcases should be restricted to reading/writing from XML. As far as I can tell there is no algorithmic reason to limit values as Strings.

For a while I can serialize/deserialize Objects to XML as a workaround.

[tkruse]

Maybe you have a better idea. What I want to model are testcases for functions with dynamic argument length (or taking a list of arguments).

As an example, consider testing a tax input function, where you submit a list of children of a person, where each child is modelled as a structure of class Child{String firstname, String lastname, Date dateofBirth}. And there is a function double childBenefit(List<Child> children, double income), where children must have between 2 and 20 entries (for some reason assume a single child gets no tax benefit, so it's an error case for the function). For the variables some conditions apply, like the firstname must be longer then 1 characters, shorter than 40 characters, only alphabetical and hyphens, .... The point is a single child entry n the list can be valid or invalid.

So what I am thinking now is that as a tester, I want to test the function giving different numbers of children. So as boundaries for the list, I would run it maybe with

• 0 valid children (failure case)
• 1 valid child (failure case, too few)
• 2 valid children (firstname 2 chars, lastname 2 chars)
• 2 valid children (firstname 40 chars, lastname 40 chars)
• 20 valid children
• 21 valid children (failure case, too many)
• 1 invalid child (firstname too short) + 1 valid child
• 1 invalid child (firstname too long) + 1 valid child
• 1 invalid child (firstname invalid character) + 1 valid child
• 1 invalid child (lastname too short) + 1 valid child
• 1 invalid child (lastname too long) + 1 valid child
• 1 invalid child (lastname invalid character) + 1 valid child ...

So using Tcases (or general combinatorial testing), how do I model this problem to generate function inputs, keeping in mind that I am seeking a generic answer that would also apply to problems with functions taking multiple lists of objects containing list structures themselves.

What I managed to do is to using Tcases on a model of a single child to create valid and invalid samples of children lists, manually create many lists following the pattern above, and then use Tcases with those lists as variable values to combine with values for salaries. But running Tcases twice this way prevents Tcases conditions to work across salaries and children. Also it generates too many valid conditions to test for all-pairs.

Ideally I would like to run Tcases just once and get a useful small number of testcases, not having to run tcases recursively. Do you have an idea for how to model the problem?

Are there maybe already publications on how to model such problems in combinatorial testing? Or is there a mailing list where such questions should be asked? (Something like this https://onlinelibrary.wiley.com/doi/abs/10.1002/stvr.1599 ?)

[kerrykimbrough]

Here's how I might model this input space.

<System name="Taxes">
    <Function name="Child-Benefit">
        <Input>
            <VarSet name="Children">
                <Var name="Count">
                    <Value name="0" failure="true"/>
                    <Value name="1" failure="true"/>
                    <Value name="2" property="children"/>
                    <Value name="20" property="children" once="true"/>
                    <Value name="21" failure="true"/>
                </Var>
                <VarSet name="Child" when="children">
                    <VarSet name="First-Name">
                        <Var name="Length">
                            <Value name="Below-Min" failure="true"/>
                            <Value name="Valid"/>
                            <Value name="Above-Max" failure="true"/>
                        </Var>
                        <Var name="Chars">
                            <Value name="Invalid" failure="true"/>
                            <Value name="Valid"/>
                        </Var>
                    </VarSet>
                    <VarSet name="Last-Name">
                        <Var name="Length">
                            <Value name="Below-Min" failure="true"/>
                            <Value name="Valid"/>
                            <Value name="Above-Max" failure="true"/>
                        </Var>
                        <Var name="Chars">
                            <Value name="Invalid" failure="true"/>
                            <Value name="Valid"/>
                        </Var>
                    </VarSet>
                </VarSet>
            </VarSet>
            <Var name="Income">
                <Value name="TBD"/>
            </Var>
        </Input>
    </Function>
</System>

[kerrykimbrough]

Are there maybe already publications on how to model such problems in combinatorial testing?

Sorry, I don't keep up with the literature. Maybe I should! The article you linked to looks interesting.

[kerrykimbrough]

As far as I can tell there is no algorithmic reason to limit values as Strings.

I think you're right.

I've generally thought of a<Value> as identifying an equivalence class, not a specific concrete input -- a name of a value, not the value itself. For example, Below-Min, etc. One benefit is that, when composing concrete test cases, you can choose different members of the Below-Min class in different test cases. Although assumed equivalent, that injects "gratuitous variety" that sometimes exposes unexpected faults.

But I see you are coming from a different perspective.

[tkruse]

Thanks for modelling.

One small issue remains though, your modelling is not minimal. Assuming for Child that we want to test multiple valid equivalence classes, we'd need one testcase per equivalence class.So If there were 20 equivalence classes for valid values, we'd need 20 tests. However, since the function is able to take 20 children, so all 20 equivalence classes for a child could be run in a single testcase (assuming we don't need to check function results for individual children).

This somehow gave me an idea. Instead of Using a Bean field like

@Var(min = 2, max = 20)
List<Child> children;

I can maybe model this instead as something like:

@Var(min = 2, max = 20)
TcasesList<Child> childrenClasses;

to indicate at a glance that a list structure is intended, and to make this way of testing lists more standardized. TcasesList can be an abstract class requiring subclasses to have a method to fill up values if needed. The annotation can specify things like whether duplicates are valid, and whether valid equivalence classes need to be tested in isolation.

Still I would like there to be a way to run my library with just an annotated List, and to do that without serializing and deserializing List values for TCases. I am not sure how much effort it would be though to extract all the String-based logic and validation to the XML reading/writing classes of Tcases.

[kerrykimbrough]

One small issue remains though, your modelling is not minimal... If there were 20 equivalence classes for valid values, we'd need 20 tests.

Yes, Tcases would generate those tests. And yes, they could be reduced to a single test using a list of 20 different Child combinations. In practice, that's exactly what I do, when I (manually) convert Tcases results into concrete, executable tests.

It would be better if Tcases could do this reduction of collection member cases automatically. But things get complicated. Depending on the specific SUT, there can be various sorts of constraints how such collections are composed. It was hard for me to see a general way to express such constraints, so I haven't really given it much more thought.

[kerrykimbrough]

I think the question of how to generate collections of combinations is separate from the original issue about the restriction of Var values to String. Agree? If so, you can add it as a new issue.

I assume you think that resolving only Var values is still worth doing. True?

[tkruse]

Feel free to reorganize any issue. I think generating collections of combinations might be out of scope for tcases, because it would require too much work (at least would not be able to contribute).

To me it feels that it's still worth dropping the String restriction of Vars to make Tcases a most versatile library, and I feel that the required changes would be manageable, with the main pain point probably being writing additional tests with non-String values (because all your tests were based on XML).

[kerrykimbrough]

Resolved by PR #39