Memory leak

[Quantum1248]

When I'm encoding video sometime one of the av1an process start using more and more memory(16GB+) resulting in a system hang. I didn't find a way to replicate the issue consistently, but it happens fairly often(I think about 50% chance for an encode with 300 scenes). When it happens the av1an process start consuming ram very fast, so I think the reason is that it's getting stuck in a loop that create object or something like that. Last time that it happened I used the following command:

av1an -i in.mkv -enc aom -w 6 -v "--bit-depth=10 --cpu-used=5 --end-usage=q --cq-level=30 --threads=4 " --boost --boost_range 20 --boost_limit 10 -a "-c:a libopus -b:a 96k " --log log -o out.mkv

[master-of-zen]

@Quantum1248 can you provide screenshots? what system config, etc.

[Quantum1248]

@Quantum1248 can you provide screenshots? what system config, etc.

I'm using Manjaro linux, python 3.8.3 and the latest version of av1an-git from AUR. I even tried to clone your repo but the problem keep happening even with the latest version. I thought it was something related to the --boost param but yesterday it happened even without it so the problem is somewhere else. I tried to debug the code myself but I didn't find anything noteworthy. The only thing that maybe(and this is a big maybe) could be the cause is that inside the encoder while true(row 761 of av1an.py) every thread call the counter.update function without synchronization and the subsequence call to tqdm_bar.update() is not threadsafe and cause the problem. However I never used python before, so probably I'm completely wrong. About the screenshot that you asked, do you want a screenshot of the process in the task manager? Or did you mean something else?

[master-of-zen]

Multiprocessing Manager, which is used for counting frames from multiple threads is thread-safe, as I know it's never caused any trouble. Yes, the screenshot of task manager. What was the source, at what stage does ram usage go up? mb attach .log file

[Quantum1248]

Next time that it happens I will post the screenshot and log. So far it happened with many different video always during the encoding at random %(yesterday it happened at about 90%, but some days ago at about 30%). However if it never happened to you probably is something with my setup that is wrong and not the program, because it happens quite often to me.

[Quantum1248]

Two things that I forgot to mention: the process that use the memory is an av1an process, and i'm using the latest version of aom-git from AUR

[Quantum1248]

Ok it happened again, here are the screenshot:

And the log: log.log

If you want something else just ask, i have suspended the process at the moment.

[master-of-zen]

Try copy current git and use it instead, I'm not creator and maintainer of aur package.

[Quantum1248]

Try copy current git and use it instead, I'm not creator and maintainer of aur package.

I already tried that, but i had the same problem

[master-of-zen]

Ok, so it was ffmpeg printing infinite amount of same error, which is written to buffer