Split between Unit Test and IntegrationTests

[dgerod]

I think that we should be able to execute most of MORSE tests without Blender, main reason is we should test our functionality independently.

Let me spit tests two categories, maybe you don't like the names but I use them to explain the proposal:

• Unit Tests, these tests are done to check MORSE components functionality without external dependencies, for that need to create mocks and stubs.
• Integration Tests, tests that must be executed with Blender and are used to check if the system is running correctly. Typically these are from user point functionality.

An example for what I am calling Unit Tests (UTs) is testing that odometry sensor is providing coordinates in a correct way. For that it is not needed to use Blender, we must define input to the test as the ones provided by Blender and check outputs as we expected. Using this type of test should avoid #774 issue. In addition, it will reduce the amount of time to execute tests as most of Integration Test (IT) should not be used all the time, these tests consume a lot of time.

The process should be:

• every time a change is done in the code, the Unit Tests related to this change must be executed without errors;
• after a commit to the master repository, Mr. CI will automatically execute all the Unit Tests and the Integration Tests. At this moment this is was Mr. CI does but only for Integration Tests as there are not tests without Blender.

[severin-lemaignan]

@dgerod while I agree with the underlying rationale, this proposal raises several practical issues (which relate to the bigger picture 'MORSE without Blender' or 'MORSE with other 3D engines'): right now we need Blender at several different stages, the first one being the 'Builder' stage: we need the Blender API to convert the MORSE Builder scripts into actual simulations -> even if we could test some functionalities without starting the game engine, we would probably still need Blender to create the simulations (well... as a matter of fact, no always: the odometry sensor, for instance, does not have any dependency on Blender...).

Could you write a sample 'odometry sensor' unit-test, to see how it looks like?

Btw, I did work some years ago on properly separating Blender dependencies from the rest of the code, see this branch: https://github.com/severin-lemaignan/morse/tree/byebyeblender that implement a mock 'pyengine' that completely replaces the 'blender_engine' -- it works, but does not do anything useful as I did not implement any code to build the simulations from the Builder scripts.

[dgerod]

Although it would be cool that we could plug other engines, my point is only about testing. So, we could focus on mock Blender stuff without creating an abstraction layer for the complete engine. Of course, it could be possible that we need to abstract some parts of Blender API.

I will create few simple tests for Odometry to check my idea, but I need to check MORSE structure and look for a mock library for Python. I never use mocks in Python... in C++ I use GoogleTest. Do you have any advice? And thanks for sharing the branch, it is quite useful.

Other point that we should discuss later is how CI could run MORSE in different OS. If we would like to guarantee that that MORSE works in different OS we should test them automatically. And I would like that works on Windows in addition to Ubuntu at least... I know that this is not so easy. At work we use several VMs that are called by Mr. Jenkins, I don't know if this approach is possible with the free CI account that is used by MORSE. Maybe it is better to add another issue to discuss this point.

[severin-lemaignan]

@dgerod for CI on Windows, I've been using appveyor quite a few times. Works exactly like travis (same level of github integration). We could definitely do that.

[dgerod]

Thanks @severin-lemaignan for the information.