Why the Lookup-Matches assertion returns error even if it is not supposed to ?

Seddryck / NBi

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile your test suite. Just create an Xml file and let the framework interpret it and play your tests. The framework is designed as an add-on of NUnit but with the possibility to port it easily to other testing frameworks.

http://www.nbi.io

Apache License 2.0

106 stars 37 forks source link

Why the Lookup-Matches assertion returns error even if it is not supposed to ? #701

Closed Davijde closed 1 year ago

Davijde commented 1 year ago

Hi All, I'm trying to build a test using the lookup-matches assertion, based on the documentation:

If a row from the candidate result-set has no matching key in the reference result-set, it’s not considered as an issue for the test and this row will be considered as successful.

Anyway, when I execute it trough Azure DevOps I get a failure with the following message

Am I missing something ?

This is the test template I'm using

Thanks in advance for any useful feedback :)

Seddryck commented 1 year ago

I was a bit by surprise by my own documentation when reading

If a row from the candidate result-set has no matching key in the reference result-set, it’s not considered as an issue for the test and this row will be considered as successful.

and the code confirmed my doubts

https://github.com/Seddryck/NBi/blob/9c01075681d4532137b6bfbbf06cbfae20e6fd93/NBi.Core/ResultSet/Lookup/LookupMatchesAnalyzer.cs#L72-L73

Clearly sounds that the documentation is not aligned with the code ... it means that the truth is in the tests! Unfortunately, the tests are giving following answer: the code is right and the documentation is wrong. We've a specific test checking that candidate rows without matching keys in the reference should be reported as violations.

https://github.com/Seddryck/NBi/blob/9c01075681d4532137b6bfbbf06cbfae20e6fd93/NBi.Testing.Core/ResultSet/Lookup/LookupMatchesAnalyzerTest.cs#L110-L119

Seddryck commented 1 year ago

I'll be waiting your feedback and especially your use-case for considering that candidate rows without matching reference keys should be marked as successful. I can always change my mind!

Davijde commented 1 year ago

Hi Seddryck, sorry for late response, my intent was to have a test that checks the following:

given two dataset A and B, all rows from A that have a Key match with B must have equal Value too. All non matching rows we don't care.

In the example above, only the two matching record would be compared, all others would be ignored and therefore the test would be successful.

Seddryck commented 1 year ago

Ok, I'll apply a fix in this case to align with the documentation.