Planner tests

[perezp]

I open this issue to remember that there is a Plan method in the Planner in the master branch that, to the best of my knowledge, is not tested.

[szenzaro]

The plan procedure is not deterministic due to optimization. Given an AAM as parameter the output depends on the optimization process. The output can change in different ways:

• the number of output plans
• the assigned hosts for each module (this also depends on the offering repository in the discoverer)

It would be nice to have an automatic test, but how can we design a proper test? What exactly should be tested?

I understand that having the single components in charge of the planning process (matchmaker, discoverer, optimizer) tested cloud not be enough. But I am willing to produce some test. Any suggestion would be appreciated.

[ahgittin]

Huh? Are you saying you can't make any assertions about the output of the planner?

If you can't, then how is anyone else supposed to use the planner's output?

[javicubo]

Definitively we should test the output of the planner with some testbed.

[rosogon]

@szenzaro : You can make the optimizer give a deterministic output if you inject to the planner a "customized" optimizer (not the real one) that gives the planner the results you want.

[szenzaro]

I agree with you all that test are needed. I am wandering what kind of test is more suitable for the plan method since it is a sequential process of already tested components (matchmaker and optimizer).

@rosogon yes I may use a "customized" optimizer but that will not test the real plan process.

@ahgittin Some assertion are meaningful and I will provide them as tests ASAP. For example assertions on the structure of the ADP generated by the optimizer through the plan process. Another possible test is the path coverage. I think I will start providing them.

[MicheleGuerriero]

@szenzaro just a comment, maybe I'm wrong, but why you want to test the structure of the ADP generated by the Optimizer? Shouldn't be the optimizer itself to realize this kind of test? As you say, the Planner is a sequential process of already tested other components, so maybe you should test just this process. Testing the Optimizer output structure would mean testing the Optimizer process, am I right?

[szenzaro]

@MicheleGuerriero I agree with you but since the plan method is an "atomic" sequence of already tested sub component algorithms (matchmaking, offering discovery and optimization) what are you suggesting to test? Testing the structure of the ADP would not be exactly testing the optimizer. I would expect that optimizer tests are related to the "value of the optimized solution" (e.g. given another assignment of hosts for the AAM the score of it is less or equal to the produced output).

[ahgittin]

Give it some input similar to what you'd expect a user to provide, and then check you get sensible outputs like what a user would want.

Tests aren't about showing optimality but more mundane essential things:

• ensuring that things aren't obviously wrong,
• getting early warning if something breaks in the future, and
• (very important) showing people how it is intended to be used.

[szenzaro]

Ok. We will provide these tests. Thank you.

[jsoldani]

@perezp we have added the test for the plan method in PR #208

Could you (or anybody else) please review it?