Thesedays I encounterd strange case where Octane benchmark shows incorrect result in case of non-deterministic mode.
Here is a simple example case.
ex )
minIteration = 4;
device_A runs 4 times in 1sec.
However, device_B runs 3 times in first iteration and couldn't satisfy minInteration and runs 7 times in additional iteration. Therefore, its final score becomes 5 in this case.
device_A : 4/1 => 4
device_B : 10/2 => 5
I think this is unexpected situation since device_A runs faster for 1st iteration. device_A is better than device_B but deterministic mode show devices_B is better.
Can someone make it clear why Octane uses non-deterministic mode by default for some test cases?
Dear folks.
Thesedays I encounterd strange case where Octane benchmark shows incorrect result in case of non-deterministic mode.
Here is a simple example case.
ex ) minIteration = 4; device_A runs 4 times in 1sec. However, device_B runs 3 times in first iteration and couldn't satisfy minInteration and runs 7 times in additional iteration. Therefore, its final score becomes 5 in this case. device_A : 4/1 => 4 device_B : 10/2 => 5
I think this is unexpected situation since device_A runs faster for 1st iteration. device_A is better than device_B but deterministic mode show devices_B is better.
Can someone make it clear why Octane uses non-deterministic mode by default for some test cases?
Thank you in advance.