Theories: Enhancement Autogeneration of datapoints

[rmahnovetsky]

I have not be able to find a way for theories to automatically generation datapoints for me. I have read about a agitar runner but I cant find much doco about it. Atm I have been creating data points or using the parameter supplier annotation. But it would be nice for theories to have a data generator out of the box. Could the datapoints generation be smart like how PEX(http://research.microsoft.com/en-us/projects/Pex/) can find boundary conditions?

[bloritsch]

The closest thing is to use the @DataPoints (note the plural) annotation. This lets you define a method that returns an array of the datapoints you want to use in your @Theory. Example:

@DataPoints
int[] generateFibonacciSeries() {
    // you can generate for real, but return the series as a array of the same type.

    return new int[] {1,1,2,3,5,8,11};
}

[pholser]

I think the point of exposing ParameterSuppliers as an API might be so that JUnit doesn't have to offer an out-of-the-box supplier. Hard to imagine a "default", one-size-fits-all JUnit ParameterSupplier -- so I'd be inclined not to try to provide one (other than the already-available internal AllMembersSupplier that scrapes @DataPoint(s)).

[dsaff]

I would love to see this come back to life. I was a driving force behind the Agitar runner, but the IP is locked up in their current holding company. In my opinion, a good datapoint-generator would be a gold trophy of the junit.contrib project.

[pholser]

Willing to contribute junit-quickcheck (https://github.com/pholser/junit-quickcheck) once the issues regarding generics and data points are resolved.

[dsaff]

@pholser, nice! Can you remind me what the next steps on the generics/datapoints front are?

[pholser]

64 is the relevant issue. Required changes to core JUnit were made here.

In this pull req, we discussed the idea of moving theories to junit.contrib, in order to avoid bringing additional dependencies into JUnit proper (namely, javaRuntype and its dependencies). I created a version of theories with javaRuntype as a dependency in this pull req, and junit-quickcheck currently builds against JUnit 4.10 and this "contrib'ed" theories.

By now, my junit.contrib theories pull req may well be out of date with any changes that have happened to the theories stuff since JUnit 4.10.

The tricky part about solving #64 is that we need to be able to decide, reliably, whether a type that involves generics can be considered a subtype of another type that involves generics. For example, if a theory parameter is of type List<? extends Number>, we should admit data points of type List<Integer>, but not List<String>. javaRuntype is very helpful with this. Guava's TypeToken may offer the same with fewer (if any) transitive dependencies.

Anyhow, if and when theories moves out of JUnit proper and becomes a junit.contrib module, I think that'd be the time either to add junit-quickcheck as a contrib module. I think bringing additional dependencies into JUnit proper is unwise (heck, I'd even advocate getting Hamcrest out of JUnit proper, maybe in a JUnit 5).

[dsaff]

Thanks, that's a great summary. I think the main hurdle to deprecating Theories in JUnit core and pointing people at the one in junit-contrib is that I don't really have a coherent distribution strategy for contrib yet. :-( Until we have that, moving Theories will make it strictly harder to use for many developers.

In my mind, we'd need:

• A maven upload strategy for contrib
• A single page pointing people at the right downloads for both core and contrib, and an overview of what can be found in each.
• A strategy for versioning and releasing contrib.

Coming up with this might be faster if it doesn't languish on my TODO list. I'll float the idea of someone else adopting it soon on the user list, unless you want first dibs, Paul.

[rmahnovetsky]

Plus one for me. I will love to see junit quickcheck added.

[pholser]

I'll pass on dibs. I'm also open to keeping the theories runner and friends in the core, as long as issues related to junit-quickcheck can be addressed...8^)

If the hood is up on theories for 4.12, maybe we can reconsider this?

[dsaff]

Are there still blocking issues for junit-quickcheck?

[pholser]

I think once #64 is solved -- whether in JUnit proper, or in a junit.contrib form of the same, or in a separate module in the JUnit project in which a javaRuntype dependency is kosher -- junit-quickcheck is full steam ahead.

[dsaff]

Where are we in junit.contrib, then? Sorry that I'm not keeping the state swapped into memory very well.

On Wed, Nov 7, 2012 at 9:53 AM, Paul Holser notifications@github.comwrote:

I think once #64 https://github.com/KentBeck/junit/issues/64 is solved -- whether in JUnit proper, or in a junit.contrib form of the same, or in a separate module in the JUnit project in which a javaRuntype dependency is kosher -- junit-quickcheck is full steam ahead.

— Reply to this email directly or view it on GitHubhttps://github.com/KentBeck/junit/issues/10#issuecomment-10150688.

[pholser]

I'm the hold-up in contrib. My next steps there would be to get contrib's theories up to date with 4.11. Beyond that, no plans laid for releasing, keeping up with core, etc.

[dsaff]

Fair enough. Let me know if I become the hold-up at any point.

[Stephan202]

@dsaff: does this discuss affect issue #109? In particular, I'd like to know whether it's worthwhile to open a pull request for said ticket. (We have some company-internal @AllEnums and @AllBooleans annotations with associated ParameterSuppliers that I'd like to open-source. (The precise names are open to discussion, of course :))

[pholser]

FWIW, the junit-quickcheck way of handling this so far is to generate values for enum and boolean parameters at random with equal probability:

@Theory public void holds(@ForAll boolean b, @ForAll MyEnum e) { /* ... */ }

and to allow for cycling through all the values if desired:

@Theory public void holds(@ForAll @ValuesOf boolean b, @ForAll @ValuesOf MyEnum e) { /* ... */ }

[Stephan202]

@pholser: very interesting; will certainly check out that project!

Given this information, perhaps I should rephrase my original question: to what extend should #109 be addressed at all, given that there is already an open-source solution to the problem. (Such questions are also relevant in the context of #537.)

[pholser]

@dsaff https://github.com/junit-team/junit.contrib/pull/8 represents theories as of JUnit 4.11 with changes required to resolve #64, which helps out junit-quickcheck quite a bit.

Feel free to float the idea of someone championing a release strategy for junit.contrib, I imagine I'll go along with whatever's deemed best.

[pholser]

junit-quickcheck now has a 0.1 release out in Maven central. It depends on junit.contrib's version of the theories runner, which is developed here, and "sanctioned" via Git subtree inclusion in junit.contrib here.

[dsaff]

@pholser, can you link jun it a quick check in from https://github.com/junit-team/junit/wiki#third-party-extensions ?

And then I think we can consider this largely closed. Thanks!

[pholser]

@dsaff done.

[dsaff]

OK, closing, since this is well handled in a well-supported extension.