Create a test suite

[glarrain]

This is a "must" for any serious software. vpython needs this both for robustness as well as the ability to increase development speed without hurting its code quality.

[BruceSherwood]

What I've done until now before making a release is run all of the example programs that are installed with VPython (in site-packages/visual/examples), from VIDLE (to test as fully as possible the actual usage environment), and look at the visual output. This is obviously somewhat tedious, and one could imagine a test procedure that just makes sure that all of those example programs run without error. But it's hard for me to imagine (which may be a failure of the imagination) a fully automated test suite due to the importance of a human looking at the graphical output. I'd be delighted to learn of alternatives! (One could create a test suite that just makes sure that all vector operations perform correctly, or that all objects such as box or cylinder can be created, but I doubt that there has ever been a state of the VPython source code where such tests would not have passed, whereas there are often cases of subtle failures that show up when running the example programs.)

[glarrain]

Hi Bruce

As you cleverly point out, it's difficult to test software where a visualization window displays graphical output. If it's dynamic it's even worse.

In a test suite there are usually 3 types of tests (http://en.wikipedia.org/wiki/Software_testing#Testing_levels):

• unit: method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use.[1] Intuitively, one can view a unit as the smallest testable part of an application http://en.wikipedia.org/wiki/Unit_testing
• integration/functional: "constructed to test whether all the components within assemblages interact correctly, for example across procedure calls or process activations, and this is done after testing individual modules, i.e. unit testing." http://en.wikipedia.org/wiki/Integration_testing
• regression: "seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes, such as enhancements, patches or configuration changes, have been made to them" http://en.wikipedia.org/wiki/Regression_testing

What you are manually doing is performing integration tests. However, how could anyone else do that if he doesn't know what to expect? This just one of the reasons of the necessity of automated testing i.e. have a test suite and run it each time a commit is added to the repository (that's the job of Travis CI by the way).

Ideally, every single line of code should be "covered" by at least one unit test. For example, you mention "make sure that all vector operations perform correctly, or that all objects such as box or cylinder can be created". Those are exactly things that ought to be tested.

[glarrain]

I'm working on this in pull request https://github.com/BruceSherwood/vpython-wx/pull/30 (not completed yet)

[BruceSherwood]

You comment, "However, how could anyone else do that (view the behavior of the example programs) if he doesn't know what to expect?" I don't see any alternative to a VPython developer getting familiar with the example programs and knowing what to expect. It's a visual medium, not just a program with well-defined inputs and outputs.

This doesn't mean that what you're doing to test such things as installation mechanisms isn't somewhat useful, but as one says in English, it's like looking for your lost car keys under the street light. It doesn't address the real issues.

Bruce

On Sat, Oct 5, 2013 at 3:26 PM, German Larrain notifications@github.comwrote:

Hi Bruce

As you cleverly point out, it's difficult to test software where a visualization window displays graphical output. If it's dynamic it's even worse.

In a test suite there are usually 3 types of tests ( http://en.wikipedia.org/wiki/Software_testing#Testing_levels):

• unit: method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use.[1] Intuitively, one can view a unit as the smallest testable part of an application http://en.wikipedia.org/wiki/Unit_testing
• integration/functional: "constructed to test whether all the components within assemblages interact correctly, for example across procedure calls or process activations, and this is done after testing individual modules, i.e. unit testing." http://en.wikipedia.org/wiki/Integration_testing
• regression: "seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes, such as enhancements, patches or configuration changes, have been made to them" http://en.wikipedia.org/wiki/Regression_testing

What you are manually doing is performing integration tests. However, how could anyone else do that if he doesn't know what to expect? This just one of the reasons of the necessity of automated testing i.e. have a test suite and run it each time a commit is added to the repository (that's the job of Travis CI by the way).

Ideally, every single line of code should be "covered" by at least one unit test. For example, you mention "make sure that all vector operations perform correctly, or that all objects such as box or cylinder can be created". Those are exactly things that ought to be tested.

— Reply to this email directly or view it on GitHubhttps://github.com/BruceSherwood/vpython-wx/issues/27#issuecomment-25757428 .

[glarrain]

As the current problems installing wxPython show, testing that an installation works properly in several (clean) environments is essential.

Nonetheless, a test suite will certainly have many more checks that. A common rule of thumb is that there is twice as much code than what's being tested!