Discussion: new project streamsx.testing

[ddebrunner]

There are two testing mechanisms today for Streams:

• Java mock framework for testing Java primitive operators (product) - Aimed at testing a single Java primitive operator but can support an arbitrary graph of Java primitives.
• Java Application API testing (com.ibm.streamsx.topology) - Allows testing of a topology built in Java or Scala. In addition since SPL operators can be invoked it can be used to test SPL composites or an arbitrary SPL graph.
  • At some point Python support will be added too.
  • This has a general mechanism where stream output can be captured and returned to the test program (even from distributed, and bluemix support is planned) to be verified. This uses Conditions and a Condition can also be used to end a distributed application, e.g. run until this stream has at least 100 tuples, or run until the stream has this contents (ordered or unordered).

This project would build upon these mechanisms to provide testing for IBM Streams applications including:

• Utilities including integration with JUnit and other assertion frameworks
  • The above mechanisms allow assertion based testing but do not force a specific one (e.g. JUnit/TestNG), having specific integration with JUnit etc. allows tests to be written more easily.
  • Ability to run the same test easily in different modes (e.g. SPL compilation flags, standalone, distributed or bluemix, different fusing, etc.etc.)
• SPL testing support using Java/Python Application API
  • Not sure what this means, may just be more examples or utilities to help this.
  • maybe auto-generate testing skeletons based upon composite/operator definition.
• Utilities for test data
  • Generated within the test
  • From files
• Samples
• How-to guides

This is more for discussion at this point rather than a proposal, I think a new separate project makes more sense than adding to streamsx.topology.

Initial code might be JUnit integration for the two mechanisms, some based upon the way that streamsx.topology does its own testing, having the ability to run the same test against standalone or distributed.

Such a project would need community buy-in involvement, for example encouraging existing github toolkits to use the testing mechanism to test their operators. The project should probably grow organically, adding stuff as people try to use the testing and find something useful that could be added.

Thoughts?

[mikespicer]

I like the idea, we often get questions asking what the best practices are for testing operators and applications. Another area that may be better handled in a separate project is data feed simulators and the ability to record and replay feeds for testing of applications and analytics.

[ddebrunner]

bump

[ddebrunner]

Going to close this due to lack of general interest.

Will probably come back with an actual proposal with some actual code and examples.

[chanskw]

I'd like to reopen this discussion. Sorry, did not have time to look at this previously.

I like the idea of the project and what it is trying to provide. I think there is a need for it as more and more people are trying to develop toolkits and operators. I got asked 3 times in the last week on how to do this.

We also have a JUnit framework that we are using internally for testing the toolkit. The framework provides the following, which may not be available with the pure topology APIs:

• Ability to set up test environment (e.g. domain, instance, environment variables, etc.) Test environment is set up using properties files. You can configure the test environment easily just by modifying the properties files.
• Validators - test validator to validate test result
• The JUnit framework supports SPL application - where we do not need to wrapper SPL composite with topology apis. By specifying a main composite name, the framework is capable of compiling, submitting and cancelling the SPL application under test.
• Ability to run tests in standalone / distributed.

I am wondering if it makes sense to make this available as part of this testing project, and we slowly work together to have a more unified framework for testing. A lot of the toolkit testing on Github is built on top of this framework.

[chanskw]

@ddebrunner do you also have examples / documentation on how to do SPL testing using the topology APIs?

[ddebrunner]

I have been improving the testing api in topology, but comparing to above:

• In general the topology api needs to be more flexible in selecting an instance (though with Bluemix any instance can be selected) so there's nothing holding back similar functionality being added.
• Validators - the toolkit has mechanisms to validate the contents of streams, called 'Conditions'.
• The framework is focused on testing the contents of streams so the topology api is used to invoke operators that produce streams, it's not clear to me how you test a main composite without assuming some form of external effect, e.g. files, which may not work on Bluemix.
• Topology can run tests in pure-java (java only apps), standalone or distributed, and will be expanded to Bluemix as well.

[ddebrunner]

As an update there is some work in topology for supporting the same condition concept for topologies defined in Python (which also supports invocation of SPL operators), including support planned for running tests on the Streaming Analytic service.

Python & Python unittest might make an easier approach for testing SPL applications than Java & JUnit.

[ddebrunner]

As an update Python tester is supported now and includes support for running tests against Streaming Analytics service. Any test is just a standard unittest.TestCase so can reuse any functionality there. E.g. a test case can submit an application and test any external effects while the app is running or after (e.g. checking a database or a file to see if the app updated the contents). Since this is just unittest any existing mechanisms for testing external contents can be used.

For Streams artifacts the focus is on checking the contents of streams, thus not requiring the test app to write out to files to verify the behaviour.

The Java topology Junit tester with 1.7 also now supports testing against the streaming analytics service.

[chanskw]

Do we want to create the streamsx.testing repository at this time? I think having a repository will help us discuss and work on creating test guidelines and best practiese. We can start working on showing how testing should be done, create documentation and also identify potential gaps in our test stories.

[chanskw]

Alright. I created streamsx.testing. We can start contributing samples and move our test discussions over there.