It seems like a lot of the testing of VideoIO is only making sure that its behavior is consistent and not that its behavior is correct. By this I mean that the testing code compares VIdeoIO's behavior to values that were likely generated using VideoIO itself. In #270 it seems like the "right" number of frames in the test videos, and generated videos, was determined using VideoIO itself. Similarly in #271 the behavior of VideoIO was assumed to be correct, and even written in a comment. There may be other examples like this lurking in the test code.
Instead of circularly validating VideoIO with itself, maybe we should rely on the ffprobe and ffmpeg binaries to give us target performance, e.g. by writing frames to picture files and then using ffmpeg to encode a "reference" video which will be used to validate VideoIO's encoding.
It seems like a lot of the testing of VideoIO is only making sure that its behavior is consistent and not that its behavior is correct. By this I mean that the testing code compares VIdeoIO's behavior to values that were likely generated using VideoIO itself. In #270 it seems like the "right" number of frames in the test videos, and generated videos, was determined using VideoIO itself. Similarly in #271 the behavior of VideoIO was assumed to be correct, and even written in a comment. There may be other examples like this lurking in the test code.
Instead of circularly validating VideoIO with itself, maybe we should rely on the
ffprobe
andffmpeg
binaries to give us target performance, e.g. by writing frames to picture files and then usingffmpeg
to encode a "reference" video which will be used to validate VideoIO's encoding.