SuffolkLITLab / ALKiln

Integrated automated end-to-end testing with docassemble, puppeteer, and cucumber.
https://assemblyline.suffolklitlab.org/docs/alkiln/intro
MIT License
14 stars 4 forks source link

Make random tests easily replicable #832

Open plocket opened 8 months ago

plocket commented 8 months ago

We can do this with a seed and a sequence of math calculations.

  1. Generate a random number.
  2. Use that random number to do math to calculate the first answer.
  3. Use a function to return a changed version of the number. For example, hash the number and turn it back into an int.
  4. Use the new number to make the next decision.

That should be a deterministic way of repeating random input. Not sure how we can use that in a retry, but there might be a way.

[We have to output the seed to the console and report and create a step that takes a seed so that the developer can easily replicate the test themselves.]

[ Resources:

Also open to other ideas.

plocket commented 8 months ago

Code to demo using hash, though not the random number, that requires no additional library (just nodejs):

const crypto = require("crypto");

const num = Math.random();
console.log(num);  // Output this

async function hash_func(seed) {
  const encoder = new TextEncoder();
  const data = encoder.encode(seed);
  // needs some kind of buffer, typed array, or data view
  const hash = await crypto.subtle.digest("SHA-1", data);
  const hashArray = Array.from(new Uint8Array(hash));
  const short_hash_hex = hashArray
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("")
    .slice(0, 8);
  return parseInt(short_hash_hex, 16);
}

// TODO: Always remember to add 1 to the result when
// the field needs a non-zero value
const hash1 = hash_func(num).then((result) => {
  console.log("1:", result);
  console.log("1:", result%8);
  const hash2 = hash_func(result).then((result) => {
    console.log(`2:`, result);
    console.log("2:", result%8);
    const hash3 = hash_func(result).then((result) => {
      console.log(`3:`, result);
      console.log("3:", result%8);
      const hash4 = hash_func(result).then((result) => {
        console.log(`4:`, result);
        console.log("4:", result%8);
      });
    });
  });
});
plocket commented 8 months ago

The package chance seems really useful and fakes data as well. The below is deterministic:

// Load Chance
const Chance = require('chance');

// Instantiate Chance so it can be used
const chance = new Chance(12345);

// Use Chance here.
const my_random_string = chance.string();
// Print the random string
console.log(my_random_string);
// $xjncp9QNS()IW&3BWl

// Repeat that
const str_2 = chance.string();
console.log(str_2);
// %)bavhDw7WU7^#P)

// Repeat that
const str_3 = chance.string();
console.log(str_3);
1dV71&IAyZGj19]xY2