Integration test with RouDi and new API elements

[budrus]

Brief feature description

After implementing and testing all the new building blocks and public API classes, a top level integration test where they are used together with the RouDI daemon shall be done

Detailed information

For this the RouDi environment can be used

Collect the things to test here:

• [ ] All public API classes and all there functionlity is verified

• [ ] The management of shared memory resources like ports shall be tested. These resources are created and shall be destroyed when creating/destroying public API objects

[budrus]

@elfenpiff @elBoberido is the RouDi environment the way to go, or are there other options in the meantime?

[elfenpiff]

@budrus for that scenario I would not use the roudi environment and explore other options like for instance: https://robotframework.org

The reason is an actual bug: In the waitset there was once the semaphore configured as a non inter process semaphore which lead to green tests since they were all started in the same process but as soon as I started an interprocess example nothing worked since the trigger was not send.

And maybe we have even more inter process bugs which we are not aware and they are not showing up when we run everything in one process. You have to keep in mind that the roudi environment does not everything which a normal roudi does.

• The roudi environment creates and maps the shared memory into the process. In the normal case, the roudi creates the shared memory and maps it and the application opens the shared memory and maps it too, but this makes no sense for the roudi environment (or maybe we can map the same shared memory multiple times into the process space)

• What about the rights management. Apps are started with specified users/groups to handle their access. This is also not very well testable in the roudi environment.

• How should we test the termination of apps with the roudi environment. If we would enable termination the only thing the test would do would be to commit Seppuku.

I think the roudi environment is the perfect tool for our classical integration tests but if we would like to have a full blown API/system test we should mock/adjust nothing and work with the real stuff.

[budrus]

Sounds like a nice job to explore something new. We should give https://robotframework.org/ a try

[elBoberido]

@budrus yes, the RouDiEnvironment is the easiest way to test the new API. Keep in mind that each thread has its own PoshRuntime. We should also explore different frameworks for our integration tests, but it's not black and white and there are valid use cases even for the tests you want to do.

[elBoberido]

@budrus for that scenario I would not use the roudi environment and explore other options like for instance: https://robotframework.org

The reason is an actual bug: In the waitset there was once the semaphore configured as a non inter process semaphore which lead to green tests since they were all started in the same process but as soon as I started an interprocess example nothing worked since the trigger was not send.

In this case we need additional tests. Like the robot framework which automates our examples as tests.

And maybe we have even more inter process bugs which we are not aware and they are not showing up when we run everything in one process. You have to keep in mind that the roudi environment does not everything which a normal roudi does.

* The roudi environment creates and maps the shared memory into the process. In the normal case, the roudi creates the shared memory and maps it and the application opens the shared memory and maps it too, but this makes no sense for the roudi environment (or maybe we can map the same shared memory multiple times into the process space)

There is no problem to do this, since the application do not need to map the shared memory to the same address like roudi.

* What about the rights management. Apps are started with specified users/groups to handle their access. This is also not very well testable in the roudi environment.

Yes, but this is a small part of the tests. No need to use a more complicated framework for tests which are not affected by this.

* How should we test the termination of apps with the roudi environment. If we would enable termination the only thing the test would do would be to commit Seppuku.

I think the roudi environment is the perfect tool for our classical integration tests but if we would like to have a full blown API/system test we should mock/adjust nothing and work with the real stuff.

Agreed

[elfenpiff]

* The roudi environment creates and maps the shared memory into the process. In the normal case, the roudi creates the shared memory and maps it and the application opens the shared memory and maps it too, but this makes no sense for the roudi environment (or maybe we can map the same shared memory multiple times into the process space)

There is no problem to do this, since the application do not need to map the shared memory to the same address like roudi.

@elBoberido I agree with you that there should be no problem. But in the end I would argue only because we do not see any problem here doesn't mean there isn't any. Also what about the future, maybe we creating one when we finalize the shared memory split.

I agree that a simple API test can be performed by the RouDiEnvironment but if we would like to test the public API also in a way that it behaves exactly like it should be in every detail we cannot use the RouDiEnvironment.

Maybe this is something for iceoryx v2.0.

[dkroenke]

The Robot Framework offers the possibility to start/stop processes and examine their results This could be a starting point for it to create integration tests. In this case we could even reuse the existing examples to achieve that. But we need then to extend the examples for covering the full existing API which make the examples more complex.

[budrus]

Or have "special examples" that execute one functionality, do a print or whatever is needed for the examination and then do the next thing. What is your feeling @dkroenke using RouDi env or the Robot Framework?

[dkroenke]

My opinion is that the Integrationtests should stick as much as possible to the usecase of the end-user, who doesn't use the RouDi Env. Regarding the Robot-Framework it seems that the support for C/C++ seems to be limited. It can start/stop processes and evaluate the return values but this is not enough for our needs. Currently I'm evaluating the ROS 2 launch testing which seems more promising. The idea is to have the API tests in a ROS 2 package available which can be easily executed with colcon test.

[dkroenke]

Example from https://github.com/eclipse/iceoryx/pull/421 can be taken up as candidate for integrationtesting