Test Engine: Functionality Analysis

[acrm]

Overview

Now we need to practice a little bit with LINQ and File Ouptut. Also we want to create a reliable and informative Quality Assurances system for our applications development process. A good point of intersection of these interests is code analysis for our Calculator library. We can obtain information about this library functionality, process it with LINQ, form some kind of report and write it to a file.

But how can we obtain information about functionality? One way is to parse source code but it's too complicated to extract all this classes, methods, instructions and expressions information and to preserve all relations between all this entities! If we are currently stumbled with parsing our own simple javascript-like language, so we obviously don't dare to mess with C# parsing. And without parsing all our source code is just a lot of text. Text is ill-structured kind of information and LINQ is suitable for well-structured information operation. And, on another hand, even if we get somehow those structured information about statements and expressions will it help us to estimate quality of applications functionality? I doubt it. It require too high level of understanding of coding principles for our analyzer.

Well, maybe we don't need to analyse code of methods. Basically we need to know what classes with what method do we have, and do these methods implement their logic correctly. We do not need to know how exactly they implement it, it's fine to operate with them like they are blackboxes for us, we just need to know whether they implement their logic correctly. And tests can provide us such information! We need to obtain information, which methods has application and find test for each methods and get from these test information, do their logic is implemented in these methods or not.

For that purpose our test should be written in special form, each test should definesome functionality of our application. We will discuss such form later. And by now, we need somehow obtain information about our application classes and methods. An instrument which helps us is Reflection. Each .NET-library - so called assembly - contains meta-information about all its classes, methods, properties, etc. Using reflection we can extract this information (it's well-structured so LINQ will help us), find corresponding tests for methods, and form a report about test coverage and correlate this information with tests report and estimate quality of our application functionality. But for begging we will prepare the simplest report about assembly's structure.

Tasks

• [x] Create class FunctionalityAnalyzer with method PrintReport(), which collects using Reflection information about all classes in the Calculator.Logic assembly and their methods and print it to a file in hierarchical form
• [x] Add this report file to source control and upload to repository

References

Reflection

[acrm]

Optional Task: Report with Markdown

Markdown is great markup for text documents, cause it's well readable both in raw and in rendered view. We can print our human-readable reports using Markdown formatting and with .md file extension. Such report will be look-nice in simple text-editor and it will looks great on the Github!

Task

• [ ] Change report format to markdown, or if you are finished with #7 Issue Multiple-subscriber just register additional output subsystem, which use markdown formatting.

[acrm]

Optional Task: Report with XML

Markdown formatting is great for humans, but it's not strict-structured enough to make this information easy-readable for programs. And, possibly, we would like to use this reports as input data for some another analytical program.

I'm sure that you familiar with XML, at least in common words. It is possible to write all xml-tags explicitly as strings in your code, like this:

var xml = "<class name=\"" + class.Name + "\">" 
    + "<method name=\"" + method.Name + "\">"
    + "</class>";

But there is enough good libraries for more fluent xml-building, including built-in .NET classes. I strongly recommend to use XDocument.

Task

• [x] Change report format to xml, or if you are finished with #7 Issue Multiple-subscriber just register additional output subsystem, which use xml formatting.

References

Wikipedia: XML - just briefly overview MSDN: LinqToXml - just briefly overview MSDN: XDocument - focus on this Habrahabr - nice example

[acrm]

You have a few comments on your two last commits and also don't forget to pull event-system after merging #7 branch and use it for additional printers bounding.

[acrm]

Additional task: Coverage analysis

Let's make our report more informative and add information about which method is covered by which test. Assume that we feed to our report system the tests assembly as well. But how we can correspond test and method, that implements tested functionality?

One way is to extract information about all method calls from the test methods body. But it is tricky because method's body isn't a plain C# code - it is a compiled IL-code, a sequence of bytes. Another way is to mark somehow, that this test method tests this functional method. We can put methods name as part of test name, retrieve tests names by reflection and parse both names. But it makes test names complex, especially in case of multiple methods calls inside. Also it is easy make a typo in substring because compiler won't make spellcheck in method names, obviously.

Better way is too use Attributes, which are just like tags, that we put to classes, methods, fields or parameters. Their purpose is to marker some elements of assembly and attached to them some supplementary information, that can be retrieved on compilation stage or by reflection search through assemblies.

Tasks

• [x] Create TestFixture attribute to mark test-classes and Test attribute to mark test methods. Mark by them all test in Calculator.Tests project. Both attributes are simple and do not contain any additional information
• [x] Create Covers attribute, which contains two parameters: Class of type Type and Method of type string. Parameters are used to specify method of class, which functionality the current test is covers. Mark by this attribute all test in Calculator.Test project. For specifying Class and Method use C# constructions typeof() and nameof().
• [x] Collect coverage information from all tests in all testfixtures to dictionary and provide this data for Functionality report methods, that should print for each method all test, that covers it.

References

Attributes typeof nameof

[acrm]

Could you please finish first with all remarks from this issue so that we can close it?

[acrm]

It's fine, you can merge this branch.