enketo / enketo

Enketo web forms monorepo
Apache License 2.0
9 stars 15 forks source link

enketo-express/test/survey-model: reduce test flake #1307

Closed alxndrsn closed 1 month ago

alxndrsn commented 1 month ago

Occasional test failures can be seen in Survey Model tests, e.g. https://github.com/enketo/enketo/actions/runs/8701494028/job/23863630099:

1) Survey Model
       getList
         obtains the list surveys if all are active in ascending launch date order:

      AssertionError: expected [ { …(2) }, { …(2) }, { …(2) } ] to deeply equal [ { …(2) }, { …(2) }, { …(2) } ]
      + expected - actual

           "openRosaId": "a"
           "openRosaServer": "https://kobotoolbox.org/enketo"
         }
         {
      +    "openRosaId": "b"
      +    "openRosaServer": "https://kobotoolbox.org/enketo"
      +  }
      +  {
           "openRosaId": "c"
           "openRosaServer": "https://kobotoolbox.org/enketo/deep"
         }
      -  {
      -    "openRosaId": "b"
      -    "openRosaServer": "https://kobotoolbox.org/enketo"
      -  }
       ]

      at /home/runner/work/enketo/enketo/node_modules/chai-as-promised/lib/chai-as-promised.js:302:22
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

This appears to be due to timestamp collisions causing surveys.sort(_ascendingLaunchDate) to be unstable.

Creating two Date objects 1ms apart in javascript does not guarantee they will have been created in different clock milliseconds. This can be demonstrated:

const DELAY = 1;
let start, end, iterations;
iterations = 0;
const recordEnd = () => {
  end = new Date();
  if(start.toISOString() === end.toISOString()) {
    throw new Error(`Start and end matched after ${iterations} iterations`);
  } else repeat();
};
const repeat = () => {
  ++iterations;
  start = new Date();
  setTimeout(recordEnd, DELAY);
};
repeat();

On my machine, a 1ms delay will cause failure after ~10 iterations. A 2ms delay will not fail before I get bored of waiting.