search

pyannote / pyannote-audio

Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding
http://pyannote.github.io
MIT License
6.03k stars 758 forks source link

ResNet backbone vs. mask pooling #1634

Open hbredin opened 8 months ago

hbredin commented 8 months ago

It has been noticed that the 3.1 pipeline efficiency suffers from speaker embedding inference. With the default config, every 10s chunk has to undergo inference 3 times by the embedding model. It proves effective by separating the whole embedding model pipeline into the resnet backbone and the mask pooling. With this modification, every chunk only needs to be inferred one time through the backbone, bringing almost 3x speedup in my experiment. Furthermore, cache inference strategy helps a lot as well, given the default overlapped ratio of 90%.

Originally posted by @mengjie-du in https://github.com/pyannote/pyannote-audio/issues/1621#issuecomment-1918314722

Hey @mengjie-du, that's a nice idea. Would you contribute this to the pyannote.audio codebase? I tried to send you an email at the address mentioned in this paper but received an error message in return -- so I am taking my chance here.

mengjie-du commented 8 months ago

Sorry for late response. My experiment code is built upon a unwarpped version of 3.1 pipeline only for cpu, which includes two mentioned separated onnx model (ResNet backbone and the final FC). The musk pooling is implemented using Numpy. Thus, the code can't be embedded into the pipeline directly. I plan to test a version compatible with the pipeline soon. I think the key point is the line 322 in pipelines/speaker_diarization.py, where the same wave data is yielded three times. It would be more effient to yield used_mask with a shape like (spk, 1, num_frames). This adjustment would allow the model to infer all of these together without some big modifications to the control flow..

hbredin commented 7 months ago

Thanks. Will also look into this myself.

stale[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.