But: Json output differs from terminal output for ea_iid

[b-ertel]

I ran ./ea_iid on a sample file.

The output from the terminal differs to the output stored in json.

terminal output:

            statistic  C[i][0]  C[i][1]  C[i][2]

---

            excursion       6       0      19
   numDirectionalRuns       6       0      26
   lenDirectionalRuns       0       6     228
numIncreasesDecreases     106       1       5
        numRunsMedian       8       0       6
        lenRunsMedian      87       6       0
         avgCollision       6       0      14
         maxCollision       5       1      86
       periodicity(1)      17       0       6
       periodicity(2)      74       0       6
       periodicity(8)      33       0       6
      periodicity(16)       6       0      19
      periodicity(32)       6       0      11
        covariance(1)      29       0       6
        covariance(2)      17       0       6
        covariance(8)       6       0      19
       covariance(16)      10       0       6
       covariance(32)      11       0       6
          compression       6       0      15

json output:

Statistic, C[i][0],C[i][1], C[i][2] avgCollision, 6, 0, 14 compression, 17, 0, 6 covariance1, 29, 0, 6 covariance16, 29, 0, 6 covariance2, 29, 0, 6 covariance32, 29, 0, 6 covariance8, 29, 0, 6 excursion, 6, 0, 19 iteration, 0, 1, 2 lenDirectionalRuns, 0, 6, 228 lenRunsMedian, 87, 6, 0 maxCollision, 5, 1, 86 numDirectionalRuns, 6, 0, 26 numIncreasesDecreases, 106, 1, 5 numRunsMedian, 8, 0, 6 periodicity1, 17, 0, 6 periodicity16, 6, 0, 19 periodicity2, 74, 0, 6 periodicity32, 6, 0, 11 periodicity8, 33, 0, 6

Looking at the compression values one can see the difference. A bit confusing...

[joshuaehill]

I would guess that these are from two different runs, as I'm not sure how to get both types of outputs from one run.

If this is true, then that explains the mismatch. The permutation testing is non-deterministic, and the count of tests above/below/equal is not expected to be consistent from run to run.

[b-ertel]

The output is from the same run. I guess the terminal output is the correct one because the json output has always the same numbers for covariance1, covariance2, etc.

But you can easily check this for yourself.

[joshuaehill]

I see, you can force both outputs! Thanks, I didn't notice that! I essentially never use the JSON output.

In my testing, I get a mismatch on many of the results as well, so this should be easy to debug.

[joshuaehill]

I think that this is due to a fairly straight-forward problem. The results sent to the console were correct, but the results sent to JSON output were not.

Hey @celic : Exciting conclusion: The JSON output for iid testing is presently broken on the NIST ESV server. This bug could cause IID testing to incorrectly pass the IID testing in some cases where it ought to have failed.. I haven't looked, but I'd guess that the JSON reporting for these tests never worked right.

[joshuaehill]

The PR that I just posted should resolve your problem. Let me know if that doesn't work out, or if anything really exciting happens. Thanks for the report!