xUnit test project added

[FSharpCSharp]

In this process, a new DLL project was added to automate the tests in the future with the help of xUnit. Furthermore, all existing assemblies have been prepared to be fully tested (including internal classes).

The adaptations make it very easy to work with test automation in the future. This improves the code quality and special unit tests can always ensure that the existing code continues to function if a change is made.

[jhoek]

@FSharpCSharp Do you also have some tests to contribute? ;-p So far, my trade-off between bugs found and time spent has been to run some C/SIDE applications (one of which was the NAVNL localisation) through the C# interface, and later through the PowerShell interface. I understand this would not find all bugs, but it would certainly find a class of bugs that occur in real life - at a relatively low cost.

Could you please help me understand the idea behind adding this test project at this point in time?

[FSharpCSharp]

@jhoek Yeah, I got some tests on hand. I am currently working on the forms and the dataports, and for this I will work directly with test cases. That's why you took this precaution!

[jhoek]

@FSharpCSharp Ah, I see. Mind you, I'm not saying that I'm not willing to accept this change into the project, I'm just careful to accept stuff that I'll possibly need to maintain, and whose existence I may need to justify to others at some point. :-) I don't plan to add any unit tests for the existing object types, so wouldn't it be strange to have only unit tests for your forms and dataports?

[FSharpCSharp]

That almost sounds like you're afraid of unit testing. That's not supposed to happen right now. The big advantage of the unit tests is the completeness check for serious program changes. For example, imagine you accept my customizations for the Form and Dataport objects. Now, for example, you are rebuilding important points in the "Core" project. This leads to certain processes changing or new concomitants. For example, since you haven't studied my adjustments regarding the new object types in detail, you overlook the fact that there is a leak in the program that causes errors under certain conditions. If you now have unit tests, you can make sure that all existing functionalities continue to work. If it should come to a break, then you notice that already during the development phase. For this reason, unit tests are written. Microsoft itself also relies on the xUnit project. So I switched all my existing Visual Studio tests to this framework, because if Microsoft does, there must be a reason for it.

You don't have to create unit tests for your existing functionality. They make sense especially when a case as described above occurs. But I would strongly recommend you to deal with it, because you can easily and automatically avoid many mistakes in the development right from the start.

[jhoek]

@FSharpCSharp I wouldn't call it afraid. I may be wrong, but I feel unit tests make sense for finding issues in the basic building blocks of your application that either don't easily become apparent in the application-as-a-whole's output, or are difficult to test because of the nature of the output (e.g. somehow only testable through the UI). That is not the case here, I believe. Also, no amount of testing will ever find 100% of bugs - testing remains a trade-off between time invested and bugs uncovered.

Translating all this to C/Breeze:

1. Importing a NAV application into C/Breeze (text file => in-memory DOM) and subsequently exporting it back to a file (in-memory DOM => text file), then making sure the input and output are byte-by-byte the same is a low-effort test that is highly effective for real-life scenarios.
2. I could test the C# interface, but since the PowerShell interface is based on that, testing the PowerShell interface tests both layers at the same time. Only if the input and output differ do I need to dig deeper into the structure to find the source of the issue.

My focus would therefore be on an AppVeyor test script that would feed one or more localisations/add-ons through C/Breeze and compare input and output. What, according to you, would be the flaw in that approach?

[FSharpCSharp]

Thank you for describing your point of view. I now think I realize why you don't want to use these unit tests personally. But unfortunately, the reasons you gave are not conclusive and understandable for me. You don't seem to understand the purpose of unit testing. That's why I wrote down here exactly what they are for and why they are always used in development. Please read this description carefully and think about it. Nevertheless, I will use these tests in my developments because they are simply practical and make life much easier.

What is a unit test?

Unit tests (=component tests) check whether the components written by the developers work as intended. In agile methods, a very frequent execution of the component tests is aimed for quality assurance. This can only be achieved if the tests are fully automated, i.e. they themselves are a program whose execution requires no more effort than the push of a button. It is usual (but not absolutely necessary) that the test is written in the same language as the test object. In agile software development, massive use is made of component tests as part of test-driven development and refactoring.

Why unit tests?

The role unit tests play in agile software development differs significantly from the role they play in classical procedures. At the time they are created as part of test-driven development, they serve more to control the design of the software than to find defects. As part of the incremental design, they are indispensable in order to retain code that can be changed over the long term. Thus they are an important prerequisite for refactorings. Thanks to the tests, accidental changes in behaviour are detected at the touch of a button. Even if new features are added to an existing system, unit tests help: they uncover unintentional remote effects of the new functions on existing system components. In addition, they also help to document the system by clearly demonstrating intended uses and reactions of the test object.

[jhoek]

Hi @fsharpcsharp,

I don't think we disagree quite as much as you think we do! :)

I'm a big fan of any form of automated testing, I just don't see myself reaching any meaningful level of unit test code coverage for C/Breeze (even with the help of other contributors) any time soon. Without that, the benefits that you describe will not be obtained, and there's even a risk of a false sense of security when making changes to the code.

Having e.g. AppVeyor build the code and import/export/compare a number of NAV localisations/applications on each commit covers a significant number of relevant, real-life scenarios at virtually no cost (not even a proverbial push of a button)?