Add end to end testing

[PTaylour]

Moving forward we need to be able to make changes to the code base with greater confidence.

There are a few unit and integration test, but these do not provide enough coverage or clarity of expected behaviour.

So, we need to be able to describe and test how we expect videocontext to behave on a higher level.

eg

• I expect the canvas to render this frame at this time when given video in this format
• I expect the canvas to render this frame at this time when this processor is connected
• I expect this complex graph to produce the same output on these browsers.

A lot of hype around https://www.cypress.io at the moment. It could be a way to achieve this. Ideally, I don't fancy trying to build anything by hand on top of selenium or even headless chrome etc.

I'll track progress on this issue. If anyone has any thoughts or fancies getting involved, please do comment!

• [x] investigate cypressio
• [x] investigate alternatives
• [x] spike

[Sacharified]

Do you have any ideas as to how to test whether a video frame at a given time is the correct frame?

I have attempted to test this using the following method:

1. Seek to a given time within the video element (e.g. 4 seconds)
2. Use canvasctx.drawImage to draw the video frame on to a canvas
3. Use canvasctx.getImage to dump the raw pixel data from the canvas.
4. Repeat steps 1-3 and diff the pixel data.

From my testing, there is some slight variability in the pixel data between repetitions of this method on the same video, likely due to the lossless nature of the video's compression.

[PTaylour]

Yes, pretty much exactly.

I was imagining using something like this https://github.com/HumbleSoftware/js-imagediff

Where you set a tolerance https://github.com/HumbleSoftware/js-imagediff/blob/master/js/imagediff.js#L166

I've yet to test this method, but hopefully that's good enough for some useful regression testing.

Do you think there's an opportunity to do anything better?

[Sacharified]

Setting a tolerance is definitely one possibility, as long as the video chosen is always sufficiently different at each frame.

I noticed when doing some more testing that not every video displays this same behaviour, sometimes the data matches perfectly when using the method described in my previous comment. This is likely due to those videos being encoded somewhat differently to the ones I tested with previously.

Another possibility would be to use one such video as the test data, but the limitation of this approach is that you are limited to testing videos encoded in a specific manner.

[PTaylour]

We're still thinking about trying cypressio, but for now it is working well enough to be useful and is merged