MFlowInter crashes with certain combination of hpad/vpad, pel, blksize, search, overlapv and divide

[zorrozork]

Long time no see! :) This time I bring you a crash I first found on the VapourSynth port of MVTools but sure enough it's also present in this version. I didn't run tests with vpad/hpad earlier and that's why this error has been avoiding detection (default hpad and vpad is 8 which doesn't trigger the error).

The crash happens with these parameters to MAnalyze and MFlowInter: hpad/vpad = 0 pel = 4 blksize = 64 search != 6 overlapv = 8 divide = 0

I have included a zip with a test script and a source video which triggers the crash on frame 3. Also there's a green tint when the video is in YUY2, perhaps not related to this. The crash only happens in YUY2 and YV24 formats, YV12 and YV16 work. More detailed test results are in the script, but to be honest those were run with the VapourSynth version so there might be some differences, I did verify most of them with Avisynth too.

mflowinter_error.zip

[pinterf]

Thanks, dubhater mentioned a possible cause, which I have fixed and no more crash occurs. YUY2 is not handled natively, mvtools2 documentations mentiones that it requires a conversion: an older plugin function is used to convert it to a special internal planar format - this hack was used in the Avisynth 2.5 era. Interleaved2planar function, do mvtools stuff, then convert the final result by Planar2interleaved. Test build with possible fix: https://drive.google.com/open?id=1hFyMuMSzgE2y5hfzgCfh6Kkw0Sh7ss0U

[zorrozork]

Your test build seems to work, no longer crashes even with a much longer test video, once again great job! About that green tint - would it be possible to detect when video is YUY2 (and Interleaved2planar is not available) and throw an error? It's probably better to fail than to produce clearly flawed output.