Test suite for labscript

[philipstarkey]

Original report (archived issue) by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

Changes to labscript that contain unexpected bugs, risk breaking people's experiments. Worse, a mistake could adversely affect the ability to obtain scientific results, or even invalidate publications. Corner cases are often hard to catch during testing, and may persist for years. To combat this, we should implement a test suite.

I suggest that we expose the runviewer Shot class in an API and use the traces (that are reverse engineered from the hardware instructions stored in the hdf5 file) to verify that outputs still maintain the same behaviour after labscript changes.

We should also:

• have something in the test suite that detects ramps, and warns (but not fails) the test if the clock ticks (and thus the ramp evaluation points) have changed slightly.
• Determines if part of the trace is just shifted in time (so that we can determine if the shift is expected - e.g. because we increased the trigger time when fixing a bug)
• Create a comprehensive test shot that uses all standard hardware, along with expected traces (generated by hand, so that the test does not fail (or worse, pass) due to a mistake in the runviewer API) to be used before merging pull requests.
• use mercurial to pull the expected behaviour of past versions for comparison with the expected behaviour of the current version

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

• Edited issue description

[dihm]

Now that #102 is in the works and there is some more serious discussion of breaking changes (#104, #105), this effort should be revisited in earnest.