ojalgo_MultTranB_100_20 test failed

[crlalam]

In the below method if for loop ran 2 times with the same inputs, the result is different for each run. Testcase assertion was failed.

jmbench.impl.runtime.OjAlgoAlgorithmFactory.OpMultTransB.process(BenchmarkMatrix[], BenchmarkMatrix[], long)

When loop run 2 times:

When JIT enabled test-case executed very fast compared to JIT disabled. When elapsed time is very less numTrails becomes 2 (My case elapsedtime < goalduration ) as per the logic in jmbench.tools.runtime.RuntimeEvaluationTest.evaluate(). When numTrails=2 below loop is running for 2 iterations, for first run correct result was produced but it was overwritten with second run result which is not the expected result in test assertion.

In jmbench.impl.runtime.OjAlgoAlgorithmFactory.OpMultTransB() method

for (long i = 0; i < numTrials; i++) { result.fillByMultiplying(matA, matBT); }

When JIT disabled testcase took more time and elapsedtime < goalduration so this loop is executing one time only, and produced expected value. In failed case numTrials becomes 2 and passed case numTrials is one. if elapsedTime < goalDuration as per the calculation numTrials is more than or equal 2.

So before going for 2nd run initialize the result object also, then stale data will be cleared and we can expect same result for same inputs.

[apete]

I don't quite get what you're saying. Is there a problem with ojAlgo or with the benchmark code (the ojAlgo algorithm factory)?

[crlalam]

Hi Thanks for looking into it,

My issue is in jmbench.impl.runtime.OjAlgoAlgorithmFactory.OpMultTransB() { for (long i = 0; i < numTrials; i++) { result.fillByMultiplying(matA, matBT); }}

My case elapsedtime < goalduration, so NumTrails is 2, these are the results when the above loop is executed 2 times.

NumTrails : 2 i = 0 Input : 6898725549061005648 , 6739791592747404360 Result : -1702889168242773212

NumTrails : 2 i = 1 Input : 6898725549061005648 , 6739791592747404360 Result : 6439618958043074220

Expected value is -1702889168242773212 but because of the second run value was updated as 6439618958043074220 and test case failed.

I am not sure whether problem with ojAlgo or with the benchmark code.

[lessthanoptimal]

Could you explain what the input and output is? is it the sum of all the values in a matrix?

[apete]

Can you post your test code? I still grasp exactly what you do or what the problem is.

The result matrix will be overwritten/reused that's by design.

Are you testing using 1x1 matrices?

If you perform the same test using EJML and MTJ - do they also fail, or is it ok there?

[lessthanoptimal]

Not sure if this is relevant, but one thing that I think should be added to the stability benchmark is if it produces the same output when run multiple times.

[crlalam]

I ran existing test cases only. What ever the output i have added in previous comments are, System.out.print() messages i have added in jmbench.impl.runtime.OjAlgoAlgorithmFactory.OpMultTransB() method.

As peter said my main issue, is it produces the same output when run multiple times?

[apete]

I've inspected ojAlgo's multiplication code and can't see anything suspicious. Further I wrote some additional unit tests trying to mimic how the benchmark works. Haven't found a problem with ojAlgo.

Are you saying that if you turn off JIT optimisations the problem disappears?

[crlalam]

Hi , Thank you for looking in to the issue.

yes, if i turn off JIT optimisations the problem disappears.

Just i want to confirm how you mimic the test-case. Your testcase should give numTrails as 2. My case in RuntimeEvaluationTest.evaluate() method numTrails becomes 2, as elapsedTime < goalduration.

FYI my ojalgo version is ojalgo-29.44.jar.

In OjAlgoAlgorithmFactory.java

public static class OpMultTransB implements AlgorithmInterface {

    public long process(final BenchmarkMatrix[] inputs, final BenchmarkMatrix[] outputs, final long numTrials) {

        final PhysicalStore matA = inputs[0].getOriginal();
        final MatrixStore matBT = new TransposedStore<Number>((MatrixStore<Number>)inputs[1].getOriginal());
        PhysicalStore result = PrimitiveDenseStore.FACTORY.makeEmpty(matA.getRowDim(), matBT.getColDim());

        final long prev = System.nanoTime();

        for (long i = 0; i < numTrials; i++) {
            result.fillByMultiplying(matA, matBT);
    // Print hashsum of the result
    // OjAlgoBenchmarkMatrix outputstmp = new OjAlgoBenchmarkMatrix(result);
    // System.out.print(jmbench.tools.BenchmarkMatrixPrinter.hashSum(outputstmp)+" ");
        }

        final long elapsedTime = System.nanoTime() - prev;
        outputs[0] = new OjAlgoBenchmarkMatrix(result);
        return elapsedTime;
    }
}

below is the output from method

In the above code numTrails is 1 Printed hashsum of result is : -1702889168242773212

In the above code numTrails is 2 Printed hashsum of result is : -1702889168242773212 Printed hashsum of result is : 6439618958043074220

In my case numTrails is 2, so my result value is 6439618958043074220 but expected is -1702889168242773212.

[apete]

Why do you use such an old version of ojAlgo? Which version of the benchmark are you using? The ojAlgo specific code of the current benchmark wouldn't be compatible with such an old version.

If you do that test of yours with the latest version of ojAlgo and the benchmark, do you still have a problem?

[crlalam]

I will upgrade both the versions and test once, and i will get back to you.

[apete]

All you need to do is to check out the master branch from this repository...