TestCase might require custom suite to be run

[mattonem]

Another workaround for supporting custom Test suites

[mattonem]

It should be perfectly fine for a test suite to contain other instances of test suites:

That is, a test suite contains sub-instances of TestCase and TestSuite. Both individual TestCases and TestSuites understand the same protocol, so they can be treated in the same way; for example, both can be run. This is in fact an application of the composite pattern in which TestSuite is the composite and the TestCases are the leaves — see Design Patterns for more information on this pattern2 . Sunit doc

[fniephaus]

Hi @mattonem,

Thanks for the PRs and sorry for the slow response. I'm assuming that with this, smalltalkCI now works for your use cases?

Best, Fabio

[mattonem]

Hi @mattonem,

Thanks for the PRs and sorry for the slow response. I'm assuming that with this, smalltalkCI now works for your use cases?

Best, Fabio

Hey, Thanks a lot @fniephaus, I must be honest, I'd like to have it thoroughly tested before merging. I couldn't find an easy way to execute a local SmalltalkCI against a local smalltalk project... I think I will try to add a CustomTestSuite to SmalltalkCI tests. I'll update the PR asap.

[fniephaus]

I couldn't find an easy way to execute a local SmalltalkCI against a local smalltalk project...

You can just clone the repository and run smalltalkCI locally via bin/smalltalkci. That's also how most of it is developed: just run bin/smalltalkci --headfull, select an image, and wait until the image is ready. I usually use a Squeak image, because the filetree export is less verbose.

I think I will try to add a CustomTestSuite to SmalltalkCI tests. I'll update the PR asap.

Yes, that'd be great. It's nice that TestSuite follow the composite pattern. But since smalltalkCI is a service with many users, we should test critical changes like this.

[mattonem]

Hi @fniephaus, It took ages to figure that one out. It looks like Squeak test suite is very different from Pharo test suite...

• For a first, squeak test suite does not support setUp and tearDown -- yet it is possible to achieve something equivalent, overwriting the run method.
• Secondly, the squeak default test runner (the graphical one) will do the same thing as SmalltalkCIRunner currently (create its own standard test suite and inject test case by case)... Pharo, on the other hand, would take custom testSuite into account.

Those observation make the contribution less valuable for sure... it would still work, but the test runner of squeak and the test runner of smalltalkCI would have different behavior...

I guess I could make the contribution in the Pharo package, that would make more sense. Trying to align the behavior of smalltalkCi with the behavior of the default language test runner...

Any thoughts on that?

[fniephaus]

It looks like Squeak test suite is very different from Pharo test suite...

I'm afraid I was expecting that. However, the class comment of Squeak's TestSuite mentions: ... The common protocol is #run: aTestResult and the dependencies protocol

and #run: calls setUp and tearDown correctly.

the squeak default test runner (the graphical one) will do the same thing as SmalltalkCIRunner currently (create its own standard test suite and inject test case by case)... Pharo, on the other hand, would take custom testSuite into account.

This indicates that we might need dialect-specific runners in smalltalkCI. I guess calling testCase buildSuite and then adding all tests from that suite to smalltalkCI's suite defeats the purpose and won't solve your problem.

Anyway, do we have any more examples for custom test suites? It may be useful to know how they are used in the wild.

[LinqLover]

Anyway, do we have any more examples for custom test suites? It may be useful to know how they are used in the wild.

I have used custom test suites in the past for providing suite-wide isolation mechanisms / setUps & tearDowns. (To be fair, that was Python/unittest, but the basic architecture is still very similar to the original SUnit architecture.)

[mattonem]

Well, Parallel testing is definitively something that can only be achieved with custom TestSuite as it requires to run testCases in different manner. But another scenario I encountered, is dealing with 'singleton-like remote complex systems' (like a robot made of multiple actuator). Ideally, I would have:

• first TestCase named RobotStartAndStopTest which only test start and stop procedure.
• then another TestCase named RobotActionTest which would start the robot at beginning of the test suite (a set up) and execute a serie of functional tests and stop the robot (test suite tear down.)

In Smalltalk world the second test case would ideally be implemented using TestResource and not CustomTestSuite. It would ideally work like this:

But if you don't implement a composite structure to your testSuites and you aggregate all testCases into one single testSuite this actually will look like this:

Because all necessary test resources are gathered by the suite to be setup and teardown before and after running the tests, this usecase will systematically fail when executing both test cases in the same testSuite (the robot will have been stoped by the first test case).

Obviously the robot example is only one example, it could be any type of complex system (singleton like) that takes several minutes to start up (web server + microservices, a test database) -- something you don't want to start before every tests.

By the way there is an existing issue in Pharo project that is kind of related that points out the troubles we face using TestResources and Singleton...

[mattonem]

I'm afraid I was expecting that. However, the class comment of Squeak's TestSuite mentions: ... The common protocol is #run: aTestResult and the dependencies protocol

and #run: calls setUp and tearDown correctly.

I'm afraid it does not call testSuite level setUp and teadDown properly.

TestSuite>>setUp basically does not exist in Squeak... setUp and teadDown are only supported at TestCase level...

Not saying it bad or a bug, but just pointing out a major difference in the Sunit framework in both world which is driving me crazy...

for reference, the piece of code in both language that would call TestSuite>>setUp.

• in squeak:

  run: aResult
    self tests do: [:each | 
      self changed: each.
      each run: aResult].

• and Pharo:

  runUnmanaged: aResult
  self setUp.
  [ self tests
      do: [ :each | 
          each run: aResult.
          self announceTest: each.
          self changed: each ] ]
      ensure: [ self tearDown ]

[fniephaus]

Ok, in that case I suggest you focus on implementing this in the Pharo-specific packages, at least for now. Also note that you can build and test your projects with a specific branch or fork of smalltalkCI (see here for example), which may be handy to verify it works correctly for your use cases.

[marceltaeumel]

Squeak: Since TestSuite already checks the availability before and resets test resources after, it makes sense to generalize this behavior into a #setUp/#tearDown combo like the ones in regular tests.