resemble-ai / Resemblyzer

A python package to analyze and compare voices with deep learning
Apache License 2.0
2.66k stars 419 forks source link

Async and parallel processing #69

Open frankiedrake opened 2 years ago

frankiedrake commented 2 years ago

First, sorry if this question is addressed here wrong, but I should try. The original question was asked here, I described the problem in detail. In general: I have an application where some audios are being processed. There's function that embeds audio file which internally calls preprocess_wav function. When processing requests synchronously (using flask), everything is working fine, but in async-way processing (via rabbitmq using amqp-storm), my gunicorn workers are sometimes being restarted. Adding a debug points show that it breaks at the preprocess_wav call and more precisely at trim_long_silences call. What I've noticed is that there isn't high CPU or memory usage (logged it with psutil), also this even happens with the only one worker, so this tells me that probably this is not a resource usage issue but probably some blocking operations. Could you suggest me what can I do to solve this or which way I get the details on why this actually happen? Thank you.