Testing

[KGergo88]

The following topics need to be done:

Tool evaluations

• unittest - Python standard library unit testing framework
  • Link: https://docs.python.org/3/library/unittest.html
  • Decision: This will be our primary tool for the unit testing
• pytest - 3rd party testing framework for python
  • Link: https://docs.pytest.org/en/latest
  • Decision: will not use it because of bad experiences with other projects
• Coverage.py - Tool to create code coverage reports for the tests
  • Compatibility: Works with unittest and pytest
  • Link: https://coverage.readthedocs.io/en/v4.5.x/
  • Decision: Can be used for checking the unit test coverage
• Hypothesis
  • Compatibility: Works with unittest and pytest
  • Link: https://hypothesis.readthedocs.io/en/latest/index.html
  • Decision: Can be used for some tests parallel to unittest

Project structure

• [x] The issue needs to be developed on the ISSUE_#3_Testing branch
• [x] Define the project structure that has a folder for the tests and below that subfolders for unit tests, functional tests and other test relevant tools/data/mapfiles/configurations/scripts
• [x] Create a script to generate code coverage reports for the unit tests

Unit testing

• [ ] Test first the most important functions and classes:
  • [x] memoryMap.py::emma_libs.memoryMap.caluclateObjectsInSections
  • [ ] memoryManager.py::resolveDuplicateContainmentOverlap()
  • [ ] memoryManager.py::importData
  • [x] memoryEntry.py::MemEntry
• [ ] Develop full tests for the following files:
  • [ ] Files TBD...
• [ ] Further points TBD...

Functional testing

• [x] Call Emma with various wrong command line arguments
• [x] Call Emma-vis with various wrong command line arguments
• [x] Call Emma-Delta with various wrong command line arguments
• [x] Create functional tests for the Emma with test_project (this is a precondition for #5)
• [ ] Test the dependencies of arguments
• [ ] Make tests with mapfiles that have a wrong structure/syntax
• [ ] Make tests with configurations that are ill formed:
  • [ ] globalConfig.json without any configID defined in it
  • [ ] A config with an address spaces file that has an ignored memory entries that do not exist in it
• [ ] Categorisation
  • [ ] If a category does not match, we shall get a warning
• [ ] Call Emma with no map files at all and check for the error message (exit code -10)
• [ ] TBD...

Corrections needed based on the first review of unit tests

• [x] Test naming (was corrected in 641a7fb): https://github.com/bmwcarit/Emma/commit/b7bfe55c80b0f6f6696947fdf3e553b6a25d9955#r34194228
• [x] Constant naming (was corrected in e70596d): https://github.com/bmwcarit/Emma/commit/b7bfe55c80b0f6f6696947fdf3e553b6a25d9955#r34194285