Testing recipes

[vincent-zurczak]

Testing recipes is one painful job. It should be possible to write tests for such cases.

As a reminder, recipes are in charge of the life cycle control of a Roboconf instance. A good practice is that they should be idem-potent, that is to say that executing a same recipe twice should result in the same result and should not trigger any error.

Writing tests is one thing. Running them is another one. These tests should be executable by hand or automatically. The retained solution (or at least, the first to explore) is a JUnit extension, if not JUnit itself. In the same way, to not trash the test machine, we suggest to perform such tests in Docker containers.

There are two options to consider:

• Either we only run recipes in the container. No Roboconf agent. So, we simply execute a recipe.
• Or, we deploy a container with a real agent and we mimick DM instructions.

Since we are focused on recipe testing, let's only consider the first option. In this case, just like it is done in PAX-Exam, we suggest to use a method to configure the Docker container, and the other test parts will be considered as a probe that runs in the Docker container.

Here is a snippet to discuss. We assume here we have a server component in our application, managed by the script installer.

// Configure the launching of the test (Docker) container.
@Configure
public Options configureDockerContainer() {
    return Options.dockerImage( myImg )
            .dockerVolume( SRC_TEST_RESOURCES )
            .environment( new HashMap<String,String>( 0 ));
}

// What will be executed in the Docker container.
@Test
public void testBasicLifeCycle() {

    // Pre-conditions
    Assert.assertTrue( ... );

    // Run the recipe
    RecipeExecution re = RecipeExecution( "/bin/bash", "/start.sh" );
    re.run();

    // Post-conditions
    Assert.assertTrue( ... );

    // Idempotence check
    re.run();
    Assert.assertTrue( ... );

    // Or make verifications with a script
    Assert.assertEquals( 0, ProgramUtils.execute( verificationScript.sh ));

    // Update simulation
    DependencyChange dc = new DependencyChange( "component", "exporting-instance", "app", true );
    RecipeExecution re = RecipeExecution( "/bin/bash", "/update.sh",  dc );
    re.run();
    Assert.assertTrue( ... );
}

A method can test one recipe or several ones. This gives quite a lof of flexibility for developers and testers.

RecipeExecution will roughly mimic what an agent executes (it will most likely be the same code behind). In such a situation, we will also have to use a custom JUnit runner.

Notice that if we want to perform tests on a host system and not in a Docker container, we can simply use JUnit itself. Making them run in Docker containers is more complicated as it implies we will run a Java process that will execute the recipe annd perform verifications.

[vincent-zurczak]

@cdeneux @ykouki @rpignolet What do you think?

[rpignolet]

IMO, we must use Docker in order to have the same test behavior on every system (at least Linux) because if a recipe install some package we do not want these package on our machine and maybe we do not run the same system than the target.

The second option which is to mimick DM instruction is much better especially for testing update script with new import and remove import.

[cdeneux]

Sorry @vincent-zurczak, I don't understand very well what you want to do with this issue. My requirement is to be able to create a unit test (based on JUnit for example), fully integrated with Maven, to test the deployment of my Roboconf application that I'm writing.

IMO, first, I should be able to create a DM instance initialized with my Roboconf applications:

@Rule
public final xxxxx roboconfDM = new RoboconfDM()
                                       .newAppTemplate("appTmpl1").newAppTemplate("appTmpl2")
                                       .newApp("app1", "appTmpl1").newApp("app2", "appTpml2")
                                       .newAppBinding("app1", "app1-import-ref", "app2");

And, in the 2nd step, I should be able to play with instances lifecycles:

@Test
public void testDeployAndStartAllInRightOrder() {
   roboconfDM.getApp("app2").deployAndStartAll();
   roboconfDM.getApp("app1").deployAndStartAll();
}

@Test
public void testDeployAndStartAllInInversedOrder() {
   roboconfDM.getApp("app1").deployAndStartAll();
   roboconfDM.getApp("app2").deployAndStartAll();
}

@Test
public void testDeployByInstance() {
   roboconfDM.getApp("app2").getInstance("container-bootstrap-vm").deployAndStartEverything();
   roboconfDM.getApp("app2").getInstance("container-cons-vm").deploy();
   roboconfDM.getApp("app2").getInstance("container-cons-vm").start();
   roboconfDM.getApp("app1").deployAndStartAll();
}

[vincent-zurczak]

@cdeneux This is already possible if you use a real IaaS and agents and so on. There is nothing to develop to do that. Everything already exists.

This is issue is about providing something specific about recipes, which is the hardest part of a Roboconf application.