Crash in MVTools

[zorrozork]

I found a very specific combination of operations and parameters which triggers a crash in MVTools. The error is most of the time "MFlowFps: cannot work in reentrant multithread mode!", sometimes it says something about "integer division by zero at module mvtools".

I have attached a script and a small source video which triggers the error every time at frame 5 when stepping though the frames from the beginning. Making some modifications to the parameters the error can be triggered at frame 6 instead, or at frame 1. There are more detailed test results in the script comments. I tried to make a smaller script but it seems to need all the operations there, among other things two runs of RemoveGrain. It is probably depending on the RemoveGrain implementation as well, the one I have installed now is the latest RgTools.

The script uses an SSIM plugin with SSIM_FRAME-function, it can be downloaded here.

My system is Windows 10, Avisynth v.2.6.0 ST, 32-bit, MVTools version is the latest v2.7.34. mvtools-bug-2018-11-11.zip

[pinterf]

Thanks. I could reproduce it under classic avisynth but not with avisynth+ but it may only be a random lucky coincidence.

[zorrozork]

Yeah, that bug must be ultra rare, it needs very special conditions to occur.

[pinterf]

Could you please have a look at this one: the fix and some other interesting additions https://drive.google.com/open?id=1KRn4VUAgoLgwpNf4y0yUdp-Y9a38Adeu If it survives your testbench I'll release it

[zorrozork]

No longer errors with the test script. And I really like the other interesting additions as well... I was running an optimization task, about 17000 iterations out of 50000 finished while I replaced the latest MVTools in. Immediately the optimizer started finding new pareto front results (the red dots) so it's definitely faster than before!

The error in question was found while running this kind of task but it took 121730 tests to find it. I will report if I find another one... might take a while though. :)

[pinterf]

Ok, released. It should definitely fix that problem. Because of an internal 32 bit overflow, interpolated SAD values could reach negative values, at a specific point the negative SAD just zeroed out a sum in a divisor and resulted in divbyzero error. Nevertheless that divbyzero was only the peak of an iceberg; the negative SAD values just made the vector optimization in those cases wrong. My fix now is using a lower threshold for establishing the limits of safe 32 bit internal calculation, above which I should switch to a 64 bit (and thus slower) intermediate computing.

[zorrozork]

Thanks again for the fix and other niceties. I wonder how common those negative SAD values were, do you have an idea what kind of parameter combinations would cause them (other than those in the test script)?

[pinterf]

Big blksize e.g. 32, small but nonzero overlap, e.g. overlapx 4 overlapy 0, and very large individual sad values (very different areas in block comparison). Hard to tell exact numbers, if needed I can calculate more examples for you..

[zorrozork]

No don't bother, I will run more tests anyway and find out if there's a new optimal parameter combination after the fix.

[pinterf]

BTW. In your test avs script this line searchClip = RemoveGrain(orig, -1).RemoveGrain(24) # both RemoveGrains are needed for the crash caused a bit of mess in my head, did not noticed that parameter -1 means that none of the planes are filtered, the resulting clip is considered to have an undefined garbage content. (Which was mostly empty zeros for AVS 2.6 and old video cache content under AVS+) I just wanted to make a speed comparison between classic avs 2.6, avs+ x86 and x64 on my AVX2 i7 and the undeterministic search clip caused inconsistent results. Otherwise x64 won (+40% fps) after removed the -1 value

[zorrozork]

I was wondering what function that -1 has. I now realize it's meant to be used for one of the planes only for some speedup. It shouldn't have been in the script but I guess it was a lucky error because without it I wouldn't have found this issue. :)

Another idea popped in my mind: would it make sense to add a switch to force the 64 bit computing for all the calculations? It would be then easy to make a test script which finds out any differences in 32 bit and 64 bit processing (I'm assuming they should give identical results). Obviously this wouldn't have much use outside testing.