Provide regression tests (in nsiqunittest/nsiqcppstyle_unittest.py) for NsiqCppStyle engine internals

[mosherubin]

Background

NsiqCppStyle is a serious Open Source software project, intended for general use. Every time a change is made to the engine code, there is a real risk of breaking the existing implementation and/or features. NsiqCppStyle requires high-quality and reliable regression tests to ensure that changes to the existing engine code base won't affect the existing features of the project.

As expected of a well-designed product, NsiqCppStyle has a dedicated unit/regression test source file: nsiqunittest/nsiqcppstyle_unittest.py. The file defines a class unitTest with member functions, each one representing a unit/regression test. This script file currently has a small number of tests:

Test Name	Description
unitTest.__testFunctionSpecifier	Internal utility function used by the testIgnore function
unitTest.testIgnoreFinalFunctionSpecifier	Verifies that the final token is ignored
unitTest.testIgnoreOverrideFunctionSpecifier	Verifies that the override token is ignored
unitTest.testIgnoreNoexceptFunctionSpecifier	Verifies that the noexcept token is ignored
unitTest.testGetPrevMatchingLT	Verifies lexer.GetPrevMatchingLT() works correctly for simple variable definition/declaration statement using '<...>'
unitTest.testGetPrevMatchingLTWithInner	Verifies lexer.GetPrevMatchingLT() works correctly for variable definition/declaration statement using nested '<...>' pairs, no occurrence of adjacent LT or GT
unitTest.testGetPrevMatchingLTWithInnerOnes	Verifies lexer.GetPrevMatchingLT() works correctly for variable definition/declaration statement using nested '<...>' pairs, has occurrence of adjacent LT or GT
unitTest.test2	Verifies all tokens can be iterated using GetNextToken
unitTest.test3	Verifies all tokens can be iterated using GetNextTokenSkipWhiteSpaceAndComment
unitTest.test4	Same as test3 with console level set to console.Level.Verbose
unitTest.test5	Verifies the first token of a complex definition/declaration has type ID and expected value

It is clear that these tests are the proverbial "tip of the iceberg", and that many more such tests should be added to ideally provide complete coverage of internal engine code.

Request

Effort should be invested to write as many targeted unit/regression tests for NsiqCppStyle internals as possible. As more people submit code changes, having a rich set of regression tests will give us the confidence needed to make changes without breaking the existing code base.

The writing of such tests should be an ongoing task: every time any contributor (or the maintainer) touches a section of NsiqCppStyle engine code, time should be allocated for enriching the set of unit/regression tests, all related to the section/features modified.

Similarly, while implementing a new feature, should a contributor break the product during development, s/he should stop and write the necessary unit/regression tests to never allow that break to happen again.

IMHO, this should be a high-priority task as soon as the current pull requests have settled and are merged.

[mosherubin]

Considering the numerous different functional areas in the engine (e.g., tokenization, iterating the token list, verifying lexer member functions and fields), we might want to have a separate unit test source file for each area, rather than lumping them all together into a single source file. Tests should be inserted in the appropriate unit test source file. All unit test source files should then be imported into nsiqunittest/nsiqcppstyle_unittest.py.