fschr
commented
7 years ago The only solution I have come up with is this:
#!/bin/sh
if $some_state_file does not exist:
output the first value
create $some_state_file and put the next value in it
else:
output the value in $some_state_file
overwrite the value in $some_state_file with the next valuewhere $some_state_file is not tracked. This solution is not extremely clean, but I think it solves the problem.
To create useful e2e tests, we need to write scripts that are deterministic. By deterministic, I mean that the data the scripts output is guaranteed to be a certain value based on the number of times the script has run.
Suppose we want to verify that our implementation of Student's t-Test---where the null hypothesis is that the true population mean is some value, and the alternative hypothesis is that the true population mean is less than that value---correctly outputs a confidence of 99% for some data. This relies on a script which will always---deterministically--produce data such that running the aforementioned t-Test against the data will result in a rejection of the null hypothesis in favor of the alternative with 99% confidence.
The problem here lies in the bold-faced sentence above. How can we create such a script? Suppose we know that the number of times the script is run is 20. Then the script must produce a different value each time it is run (it must somehow keep track of this) such that all the values together make up a distribution that will reject the null hypothesis in favor of the alternative with 99% confidence.
I am not sure how to solve this problem. To further complicate it, this script will be version-controlled (so it cannot be self-modifying, or rely on any state that may be altered by git).