Specific frame access order causes unlimited video cache growth (ex-propCopy causes a memory leak)

[Boulder08]

As per this post, https://forum.doom9.org/showthread.php?p=1964432#post1964432, I've noticed that propCopy leaks memory. This happens also without any Prefetch calls and QTGMC or SRestore are not actually involved.

[pinterf]

Thank you for the report, I'm gonna investigate it soon.

[Boulder08]

Thank you :) I think this issue might be related since propCopy is involved as well: https://forum.doom9.org/showthread.php?p=1964113#post1964113

[pinterf]

Well, if I replace propCopy(last, org) with Stackvertical(last,org) or Merge(last,org) then it has the same leak. Actually I removed everything from propCopy implementation and it just get the two input clips and it behaved the same way. Then I thought that I'd replace it with another two-clip-input filters and they are doing the same. Could you please check it in your environment?

[pinterf]

This is my current test

ffms2("any.mp4").Loop(1000)
org=last
QTGMC(preset="fast", noisepreset="fast", tr2=0, sigma=0.25)
Merge(last,org)

EDIT: so some function call combination inside QTGMC makes this happen. QTGMC is huge and I usually do not have that intuitive spark to point on the single line :). Anyway, earlier I have already skeletonized QTGMC at least six times, since error reports usually say that something is wrong when QTGMC is involved :), it's my unfortunate that QTGMC involves a miriad functions.

[pinterf]

Oh, please link hints to script sources. I was not aware of QTGMC / STGMC change. I was using old qtgmc and though I saw a very small gain in memory consumption it seems you seeing something different.

[Boulder08]

I'll post some links here for future reference: https://pastebin.com/aEpYnj7m

https://github.com/realfinder/AVS-Stuff/blob/a487858224a063101b9bfc1e128a0953ee6c5046/avs%202.6%20and%20up/STGMC.avsi for STGMC, renamed to avoid mixing up with Dogway's version. Does not utilize frame props, so no memory leak.

[pinterf]

What is the speed of the leak? I mean: does it comsume the memory quickly or just some kilo/megabytes for each 1000 frames? I'm testing with a 1920x1080 YV12 clip.

[Boulder08]

We are talking about some tens of kilobytes per a couple of frames or so. When running the script thru AVSMeter, it often required 30-60 minutes to fill the set memory. That made it very slow to debug the issue because I didn't know which level was normal and which one is a leakage.

[Boulder08]

In addition to that, the startup phase of the script is super slow if QTGMC/STGMC is involved. NNEDI3(field=-2) starts processing almost right away.

[pinterf]

Thanks. First link gives me 404 error

[Boulder08]

Dang, it's removed.

https://pastebin.com/aEpYnj7m contains the debug function + missing parts in the same paste.

[pinterf]

Another question. Does this script do the same like the original problem? (obviously it must reach the limit much sooner)

your input source
org=last
sh_Bob( 0,0.5 ) # this line remained from the whole QTGMC
Merge(last,org)

EDIT: which can be further simplified

source
ConvertToY8()
org=last
Bob()
Merge(last,org)

This memory growth is already present in a very early Avisynth build (2016.03.29) AviSynth+ 0.1 (r1828, MT-pfmod, x86_64) (0.1.0.0)

For myself: it is caused by SeparateFields when it is actually requesting the (n/2)th frame number from source filter. https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/field.cpp#L666 When I replaced it with n/4 or n then there was no problem.

[Boulder08]

Yes, I can confirm that even the simplest versions leak memory.

[pinterf]

Thanks. I can firmly state that it can be fixed in one day's to four weeks' time :) I'm not happy.

[Boulder08]

What I find surprising is that no one ran into it before I did.. or they did but just ignored it as some random issue.

[pinterf]

When a frame pattern is jumping to and fro (frame request pattern from source filter is something like that: 0, 0, 0, 1, 1, 2, 1, 3, 2, 4, 2, 5, 3, 6, 3, 7, 4, 8, 4, ... 44, 22, 45, 23, 56, 23...) then the internal auto-optimizing cache algorithm is confused ("Ghost cache") and will grow cache size infinitely.

This script must use propCopy(last, Interleave(org, org)) instead of propCopy(last, org)

Also see my comment here: https://forum.doom9.org/showthread.php?p=1964613#post1964613

Nevertheless, even if the Interleave() thing works, do not close the ticket. This kind of bug is still there and I'd like to heal it - if once I understand what happens with Ghosts and how it works.