Open Jamiroquai88 opened 2 years ago
Ok so after some debugging it looks like memory consumption comes from
y_pred = model.estimate_sequential(input, args)[0]
I guess because the audio is quite long. Is it possible to do some kind of chunking?
Hi, thank you for the detailed information and sorry for the delay. I was aware that the inference could take quite some memory but had not run any profiler, thank you for that. I would need some extra time to understand where the main problem is and how to solve it (if possible) and at the moment I do not have that time. I will keep the issue open to take a look at it in the future.
In the meantime, the idea of some kind of chunking is something that has been explored: Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech and TOWARDS NEURAL DIARIZATION FOR UNLIMITED NUMBERS OF SPEAKERS USING GLOBAL AND LOCAL ATTRACTORS but that is not implemented here and would require some extra effort. Folks at NTT have an implementation and perhaps you could try something in that line.
With the current version that we have here, a compromise solution could be to increase the subsampling
parameter. You will lose granularity in the output but perhaps it will not affect so much the final result depending on the scenario. We did some analysis between 50ms and 100ms in Table 4 and you can see that there is quite some impact when using fine-tuning but that was when evaluating with collar 0ms. Maybe, in a more forgiving setup, the difference would not be that large.
Hey, I have been using the codebase for quite some time and I noticed extensive memory consumption of infer.py script. In my case it consumes around 120GB of RAM, which is a huge deal - my machine has 2T of RAM but I would need to run more of them in parallel.
I tried a simple RAM profiler on per-line basis and this is the output:
So it looks like the larger consumption is coming out of
infer_loader
. Any ideas on how we could improve this?I am running on pretty long audios ~1 hour.