MvTools2 2.7.45 encodes hang when working in 4:2:2 16bit planar

[FranceBB]

Hi there, encodes with MVTools2 2.7.45 hang at different times when working in 4:2:2 16bit. I've been testing this and eventually it always hangs. I noticed this when encodes stopped working in all 5 servers of our farm here at Sky. All servers are 56c/112th AVX-512 Intel Xeon running Avisynth 3.7.1.21 x64 CUDA Enabled (although we're not making use of CUDA in the workflows that get stuck). After a few tests, we've nailed the problem down to MVTools2 as we've been using scripts that make use of it in our supply chains. The encode just hangs, namely FFMpeg is still there, with RAM allocated, it doesn't crash, but it just stops, as if it's waiting for frames from Avisynth and it's not getting any, so it just... waits forever.

[pinterf]

Hi,

yes, meanwhile I saw your post and tried reporucing it unsuccessfully. I was trying a basic QTGMC with the parameters seen in the forum: QTGMC( Preset="Slower", InputType=1 )

If the script is different from that, then I need your script. Do you have SetMemoryMax? Then I need versions/links if plugins are not the latest. Namely the ffmpeg version, and the version of all your plugins in use. I suppose you are using x64.

Parameters with which ffmpeg is called.

Input video format (width, height, format) or a sample video, a short one is enough, I'm gonna try to loop it to get a longer one.

[FranceBB]

Sure. I've sent you a Faspex via Aspera which includes:

• Avisynth Script
• Bat file
• Source

You should get an email at pinterfer@gmail.com in 18 minutes

About the version, I'm using Avisynth 3.7.1.21 x64, Zs_RF_Shared V1.154, QTGMC V3.382, MVTools 2.7.45

[pinterf]

O.K. session started. No MT is used, right? (Plus a note: the script needed jpsdr's plugin pack as well)

[pinterf]

I'm doing it without KillerSpots. What is killerspots? From where can I obtain it? (Had some search results but could not find it)

[FranceBB]

Yep the version of plugins_JPSDR is the 3.2.7 x64, Clang AVX2. About MT it is disabled as I never use things like prefetch etc. I'm old school and still live assuming that each and every developer should implement multithreading on his own without Avisynth interfering with it eheheheh. About KillerSpots, I'll provide it as soon as I get to my desk. I'm gonna grab a coffee and I'll be right there, I didn't expect you waking up early eheheheh

[FranceBB]

KillerSpots.avsi here KillerSpots.zip

[DTL2020]

" I never use things like prefetch etc. I'm old school and still live assuming that each and every developer should implement multithreading on his own without Avisynth interfering with it eheheheh."

Unfortunately internal mvtools MT is still significally slower in compare with outer MT by avisynth. That is strange but fact. It only uses significally less memory.

[pinterf]

Some sessions were successfully run. I wonder exactly how did you nailed the problem down to MVTools2. This is quite a complex script with zillion of other external plugins inside. The only connection I can see is that a script which contains mvtools is a complex script with other plugins involved.

[pinterf]

Another question: I suppose the operating system is not XP?

[DTL2020]

I have an idea about try not use internal MT in mvtools. If it fixes the bug so it may be somwhere in mvtools+avstp.dll .

[FranceBB]

The OS are Windows 10 Enterprise x64 and Windows Server 2019 x64. The fact that it's running successfully on your end makes me wonder a bit. @DTL2020 let me copy paste the script here for reference, but first how can I disable internal multithreading in MVTools?

[FranceBB]

Uhm... I actually saw Boulder on Doom9 suggesting to get rid of avstp.dll from the plugin folder and try again. I'm gonna run the test again and let you know.

[pinterf]

Win10 Pro on my side but some of the plugins and/or ffmpeg build is surely different. I avstp.dll rename, or provide mt=false for MAnalyze, MRecalculate (don't know, have to look into html docs) in all used .avsi and script text. And a weird idea: Years ago I've read on doom9 that the problem of freezing was CPU overheating.

[FranceBB]

I will do it and report again. :) About CPU overheating, that cannot be as it's a 56c/112th monster from Intel kept at constant temperature in the server room.

[pinterf]

Well, I have copied over an avstp.dll (1.04) frame=29500 fps=4.5 q=-0.0 size=N/A time=00:09:50.00 bitrate=N/A speed=0.0898x Frozen as ice. EDIT: I started a debug session with debug mvtools2 - now using avstp. Let's see what happens by tomorrow morning

[FranceBB]

Thank you for cracking this down! :D As a precautionary measure I will temporarily toss away avstp.dll from my workflows so that at least my day-to-day stuff can go on, but please let me know as soon as you find out what was the cause. :) I'm sure of one thing, though: there couldn't be a better person to take a look at this than Ferenc :D

[DTL2020]

"toss away avstp.dll from my workflows"

It will probably disables internal mt in all other plugins may be used in QTGMC() so may be better found all mvtools functions inside it and add mt=false.

[FranceBB]

Yeah well yes that would be preferable. Still in the long run I'd like it to work with avstp so this is just gonna be a precautionary measure. You have no idea how long it took me to get to this point cause at first I thought it was a networking problem and then I thought it was the storage and then a gazillion of other things 'cause our farm runs on Isilon through 10 Gbit/s + AVID NEXIS RAID6 for things like Watchfolders etc so it really took me a very long time to debug and get to this point which means that it's been a while since my remastering workflows halted. Anyway thanks for this. :) Time to get my coffee now. If we were not online I would happily offer you all breakfast. :P

[pinterf]

(Morning result: no hanging with debug build (reached 50000 frames). Probably it is too slow. Sometimes 1/10000 sec timing conditions trigger a problem. Let's retry with a release version. 4 fps. Probably I can say 'no luck either' three times sooner :)

[DTL2020]

If debug not hang it an idea to run release compiled and link with program database with profiler (i use now new AMD uProf) and if it hang with still consuming cpu it will increase the ratio of time in some place in compare with normal run. But to collect enough samples it may take long time to hang. Like 1 hour of normal run + 1 hour of hanging may collect enough samples to show increasing of time in some function. And may be with program database the profiler will show this function. Or even some narrow place like spinloop in function.

пн, 25 окт. 2021 г., 10:02 Ferenc Pintér @.***>:

(Morning result: no hanging with debug build (reached 50000 frames). Probably it is too slow. Sometimes 1/10000 sec timing conditions trigger a problem. Let's retry with a release version. 4 fps. Probably I can say 'no luck either' three times sooner :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pinterf/mvtools/issues/46#issuecomment-950589485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMGLM347K3TC2CFB2RLFDTUIT6IZANCNFSM5GQWJG4Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[DTL2020]

Or it may be possible to make a release with all release optimizations and program database compiled and linked and start debug with this release. It will not be as detailed as debug build but still allow to break after hanging and walk into hanging may be. Also this case with avstp make and idea to try simple openmp at block scan loops in MAnalyse - may be it finally be faster in compare with avs-mt. I still do not understand why avstp mt is slower when it must use better data cache locality.

пн, 25 окт. 2021 г., 10:02 Ferenc Pintér @.***>:

(Morning result: no hanging with debug build (reached 50000 frames). Probably it is too slow. Sometimes 1/10000 sec timing conditions trigger a problem. Let's retry with a release version. 4 fps. Probably I can say 'no luck either' three times sooner :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pinterf/mvtools/issues/46#issuecomment-950589485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMGLM347K3TC2CFB2RLFDTUIT6IZANCNFSM5GQWJG4Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[pinterf]

Yes, now it is running in ReleaseWithDebugInfo profile.

Comment on mt: I think only MDegrainN has avstp support; when ThSAD and ThSAD2 are the same and tr<=6, it simply calles over MDegrain1..6. avstp can still be used in other filters like MSuper, MAnalyze, MCompensate

[DTL2020]

MAnalyse definitely use mt via avstp. I tried to make different patterns scan by different threads to check if it will help to performance. It also uses MSlicer to cut workunits to different mv_search threads. So the hang may be even on .wait() if some of partial search tasks do not finishes for some reason (or end signals do not received, etc). It is good question to the bug author - Do CPU load by the process visible or not when hanging state happens ? If it hangs with zero load it may simply wait for end of threads signalling infinitely.

пн, 25 окт. 2021 г., 11:52 Ferenc Pintér @.***>:

Yes, now it is running in ReleaseWithDebugInfo profile.

Comment on mt: I think only MDegrainN has avstp support; when ThSAD and ThSAD2 are the same and tr<=6, it simply calles over MDegrain1..6. avstp can still be used in other filters like MSuper, MAnalyze, MCompensate

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pinterf/mvtools/issues/46#issuecomment-950682254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMGLM2NKYL32RGWUWSV6N3UIULFZANCNFSM5GQWJG4Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[pinterf]

Task manager is showing 0% for the process. So there is a deadlock or wait somewhere.

[FranceBB]

You can see when it hangs 'cause CPU Usage goes to 0% in the task manager

[FranceBB]

In the meantime, you should know that removing avstp.dll from everywhere worked (you can see "Workflow Avisynth_Remaster" as success ):

[DTL2020]

Some news about mt:

1. The new updated 'editorconfig' looks like working as expected and commits to github now shows only changed lines.
2. The first and still poorly designed openMP test on MAnalyse H-block scan loop (test commit to separate branch https://github.com/DTL2020/mvtools/commit/fd42f429f6b194c7ce02fa29cffa71f75797b331) shows results very close to avstp-mt and it faster on my old poor 2-cores E7500 CPU in compare with Avisynth Prefetch(2) mt. Though it still uses about 85% cpu load and avs-mt uses close to 100% but about 2.5 times more memory and still runs slower. The quick test uses fetch and release 'workspace' object at each thread run and better to create array of pointers to these objects before processing and use inside threads - will try to found way how to make it. So if avstp-mt will be long to found what is hangs it may be quickly designed openmp-mt version with close or may be even better performance.

[pinterf]

Just a progress report: some days ago, after ~30 hours of debugging, making smaller and smaller scripts, getting rid of external plugins, uncommenting and hacking the half of mvtools MSuper MAnalyze and MDegrain I successfully downloaded and modified the current avstp.dll from source. Success. But I do not understand why my modification worked. Better said: the original algorithm of "lockless queue" does not seem to have design flaws. So I have to learn. I still need many days to understand the algorithm and only then will I be able to put a proper error report. Now I'm relaxing a bit with heavy Avisynth core development but will return to it soon for sure.

[FranceBB]

No problem Ferenc, you've done a lot already. In the meantime, I'm playing the parallelism card, by encoding 250 files at the same time across the 5 x 112 threads server farm (560 threads) without avstp.dll. So... take your time, no worries. :) About the "Success. But I do not understand why my modification worked.", I had to post this meme:

[bpoxy]

Just a progress report: some days ago, after ~30 hours of debugging, making smaller and smaller scripts, getting rid of external plugins, uncommenting and hacking the half of mvtools MSuper MAnalyze and MDegrain I successfully downloaded and modified the current avstp.dll from source. Success. But I do not understand why my modification worked. Better said: the original algorithm of "lockless queue" does not seem to have design flaws. So I have to learn. I still need many days to understand the algorithm and only then will I be able to put a proper error report. Now I'm relaxing a bit with heavy Avisynth core development but will return to it soon for sure.

@pinterf, any chance you would share your modified avstp.dll? It would be nice to have a solution, even if we don't understand the reason it works :smile:

[pinterf]

I have completely forgotten about it, sorry. If nothing happens within - let's say - a month, please ping me periodically.

[bpoxy]

@pinterf: ping

[FranceBB]

pong No seriously, I forgot about this too, but I'm still running every encode without avstp.dll in my plugins folder at work. So far so good, though.

[bpoxy]

pong No seriously, I forgot about this too, but I'm still running every encode without avstp.dll in my plugins folder at work. So far so good, though.

Same here. I've put avstp.dll back a few times, greedy for that slight speed increase, but ffmpeg always hangs sooner or later.

[FranceBB]

ffmpeg always hangs sooner or later.

"Frozen as ice" as Ferenc would say ehehehehe

[pinterf]

Hi all, I'm receiving your pings and pongs, and try to arrange a time-extender for myself, nevertheless let's hope the best.

[pinterf]

Gee, I've found the modded source, but since I've tried at least twenty variations, dunno if the latest one is another working experiment or what? :) Anyway, the original avstp is still not on github so first I must create a proper repo for it, credits, changes, links...

[bpoxy]

Gee, I've found the modded source, but since I've tried at least twenty variations, dunno if the latest one is another working experiment or what? :) Anyway, the original avstp is still not on github so first I must create a proper repo for it, credits, changes, links...

Appreciate your efforts! Let me know if there's anything I can do to help.

[pinterf]

Yep, I'm in the homestretch, run the tests all day long, so far so good. Only a proper code cleanup is needed.

[bpoxy]

@pinterf, just checking in on your progress.

[pinterf]

Finally. Enjoy. Or not. I hope it works as flawlessly as it worked on my PC for many hundreds of hours :) https://github.com/pinterf/AVSTP/releases/tag/1.0.4.1

[bpoxy]

Finally. Enjoy. Or not. I hope it works as flawlessly as it worked on my PC for many hundreds of hours :) https://github.com/pinterf/AVSTP/releases/tag/1.0.4.1

Huge appreciation for your efforts! I'll start using it right away!

[FranceBB]

Outstanding work, Ferenc! Issue closed! ;)