How to implement a Precision Tolerance feature

[mlsomers]

I am trying to add a tolerance feature for a project we have (a large scale rewrite, where the new codebase is tested against the old tests, but are failing because there is a lot less intermediate rounding during calculations).

I tried to override the methods: DoCell(Parse cell, int column){} Wrong(Parse cell) Wrong(Parse cell, string actual)

However these methods are never called! (using ColumnFixture but also tried on DoFixture) I put a breakpoint inside a method that gets a result and tried to find the comparison in the stack, but either I missed it or the code is first gathering all the values before doing the comparisons... After spending some hours reverse engineering I thought it would be better to just ask where the compare magic happens?

If I succeed in adding a good working tolerance feature I'll return with a nice and tidy pull request :-)

[jediwhale]

Take a look at Cell Operators (https://fitsharp.github.io/Fit/CellOperators.html). You can write a CompareOperator that implements your own logic to compare cell values. See the CompareXxxx classes in source/fitSharp/Fit/Operators for some examples.

[mlsomers]

Thank you @jediwhale I tried to implement a CompareOperator However I must be registering it the wrong way? I added it to the Suite Configuration File and it is being loaded (it fails if I misspell the namespace or class name), however my breakpoints in CanCompare or Compare are never hit.

I also tried adding the operator in code, first in my test constructor but moved it to the Execute method because Processor was null, however in Execute() Processor is also null so I tried this:.

    public override void Execute()
    {
        if(Processor is null)
            Processor = new Service();

        Processor.AddOperatorFirst("MyNamespace.ToleranceCellOperator");
    ...

It did not fail to run but it also did not call CanCompare or Compare in my 'ToleranceCellOperator'. This is my 'ToleranceCellOperator' code so far:

    public class ToleranceCellOperator : CellOperator, CompareOperator<Cell>
    {
        private static readonly HashSet<Type> supportedTypes = new HashSet<Type>(new[]
        {
            typeof(string),
            typeof(float),
            typeof(double),
            typeof(decimal),
        });

        public static decimal Tolerance { get; set; } = 0.01m;

        public bool CanCompare(TypedValue actual, Tree<Cell> expected)
        {
            return supportedTypes.Contains(actual.Type);
        }

        public bool Compare(TypedValue actual, Tree<Cell> expected)
        {
            decimal val = StrToDecimal(actual.ValueString);
            decimal exp = StrToDecimal(expected.Value.Content);

            return (Math.Abs(exp - val) < Tolerance);
        }

        private static decimal StrToDecimal(string strVal)
        {
            strVal = strVal.Replace(',', '.');            // convert to invariant culture
            while (strVal.Sum(a => a == '.' ? 1 : 0) > 1) // strip thousands separators
            {
                int idx = strVal.IndexOf('.');
                strVal = strVal.Substring(0, idx) + strVal.Substring(idx + 1);
            }

            return decimal.Parse(strVal, CultureInfo.InvariantCulture);
        }
    }

Any hint on what the missing link is to get it to use the new ToleranceCellOperator?

[jediwhale]

I don't see anything wrong in what you did. Here's a silly little working example I just wrote. See if you can make this work for you.

Sample.cs:
namespace SampleSUT {
    public class Sample {
        public int Add(int a, int b) {
            return a + b;
        }
    }
}

MyCompare.cs:
using fitSharp.Fit.Operators;
using fitSharp.Machine.Engine;
using fitSharp.Machine.Model;

namespace SampleSUT {
    public class MyCompare: CellOperator, CompareOperator<Cell> {
        public bool CanCompare(TypedValue actual, Tree<Cell> expected) {
            return actual.Type == typeof(int);
        }

        public bool Compare(TypedValue actual, Tree<Cell> expected) {
            if (actual.GetValue<int>() == 42) return false;
            return actual.GetValue<int>() == int.Parse(expected.Value.Text);
        }
    }
}

sample.config.xml:
<suiteConfig>
    <ApplicationUnderTest>
        <AddAssembly>./bin/Debug/net5.0/SampleSUT.dll</AddAssembly>
        <AddNamespace>SampleSUT</AddNamespace>
    </ApplicationUnderTest>
    <Fit.Operators>
       <Add>SampleSUT.MyCompare</Add>
   </Fit.Operators>
   <Settings>
        <InputFolder>./tests/in</InputFolder>
        <OutputFolder>./tests/out</OutputFolder>
        <Runner>fit.Runner.FolderRunner</Runner>
   </Settings>
</suiteConfig>

sampleTest.txt in tests/in:
test@

sample
check add 1 " " 2 3
check add 1 " " 41 42

Run tests with command (this is on Linux): dotnet ~/.nuget/packages/fitsharp/2.8.2/lib/net5.0/Runner.dll -c sample.config.xml

This makes any integer compare to '42' fail, so result is one pass and one fail

[mlsomers]

Thank you @jediwhale , Indeed your example does work. But when I change it to the way I am using Fitnesse it stops working. These are the most important things I changed:

Sample.cs:

using fit;

namespace SampleSUT
{
    public class Sample: ColumnFixture
    {
        public Sample()
        { }

        public int A { get; set; }

        public int B { get; set; }

        public int Result { get; set; }

        public override void Execute()
        {
            Result = A + B;
        }
    }
}

sampleTest.txt

!*< hide stuff 
!define TEST_SYSTEM {slim}
!path ".\SampleSUT.dll"
!define TEST_RUNNER {".\Runner.exe"}
!define COMMAND_PATTERN {%m -c ".\sample.config.xml" -r fitSharp.Slim.Service.Runner %p}
*! 

|Sample      |
|A|B |Result?|
|1|2 |3      |
|1|41|42     |
|1|3 |9      |

I had to do some more stuff to get it working, like running Fitnesse.jar and copying Runner.exe to the output directory.

This is the result:

As you can see, 42 is not failing. (I did not change MyCompare.cs or sample.config.xml). Any ideas how to get it to work in this scenario?

[jediwhale]

I'll try your example and see if I can reproduce your results.

[jediwhale]

You are using fitSharp.Slim.Service.Runner in your COMMAND_PATTERN. This uses the Slim test system which does not support cell operators. You need to use the Fit test system with fitnesse.fitserver.FitServer as the runner, via the -r option in COMMAND_PATTERN or Runner node in the suite configuration file. (See https://fitsharp.github.io/FitSharp/SuiteConfigurationFile.html)

[mlsomers]

Thank you, that was it! Got it working now!

This is the class (hope someone will find it useful):

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using fitSharp.Fit.Operators;
using fitSharp.Machine.Engine;
using fitSharp.Machine.Model;

namespace DiscountTest.Helpers
{
    /// <summary>
    /// Compares values allowing a specific margin of difference between expected and actual values.
    /// This will allow for rounding differences between different generations of implementations of the same functionality.
    /// </summary>
    public class CompareWithTolerance : CellOperator, CompareOperator<Cell>
    {
        private static readonly HashSet<Type> supportedTypes = new HashSet<Type>(new[]
        {
            typeof(string),
            typeof(float),
            typeof(double),
            typeof(decimal),
        });

        public static decimal Tolerance { get; set; } = 0.02m;

        public bool CanCompare(TypedValue actual, Tree<Cell> expected)
        {
            return supportedTypes.Contains(actual.Type);
        }

        public bool Compare(TypedValue actual, Tree<Cell> expected)
        {
            if (actual.ValueString.ToLowerInvariant().StartsWith("error"))
                return false;

            if (string.IsNullOrWhiteSpace(expected.Value.Content))
                return true;

            if (expected.Value.Content.Any(char.IsLetter))
                return string.Compare(actual.ValueString, 0, expected.Value.Content, 0, int.MaxValue, true, CultureInfo.InvariantCulture) == 0;

            string filteredActual   = actual.ValueString.Replace("%", string.Empty);
            string filteredExpected = expected.Value.Content.Replace("%", string.Empty);

            decimal val = StrToDecimal(filteredActual);
            decimal exp = StrToDecimal(filteredExpected);

            bool ret = (Math.Abs(exp - val) < Tolerance);
            return ret;
        }

        private static decimal StrToDecimal(string strVal)
        {
            try
            {
                strVal = strVal.Replace(',', '.');            // convert to invariant culture
                while (strVal.Sum(a => a == '.' ? 1 : 0) > 1) // strip thousands separators
                {
                    int idx = strVal.IndexOf('.');
                    strVal = strVal.Substring(0, idx) + strVal.Substring(idx + 1);
                }

                return decimal.Parse(strVal, CultureInfo.InvariantCulture);
            }
            catch (Exception ex)
            {
                throw new Exception(string.Concat("Parse string was \"", strVal, "\"."), ex);
            }
        }
    }
}