google / lyra

A Very Low-Bitrate Codec for Speech Compression
Apache License 2.0
3.83k stars 355 forks source link

MOS value is very low in WebRTC p2p call #45

Open pyu1538 opened 3 years ago

pyu1538 commented 3 years ago

I have integrated Lyra into WebRTC and tested it on two Pixel 3 phones(Android 11), the MOS(measured by Malden red box) of p2p call is only about 2.2, it can't reach the VoIP call standard, do we have a plan to optimize its quality?

aluebs commented 3 years ago

It is awesome that you integrated Lyra into WebRTC for real-time calls. However, those MOS scores seem very low compared to what we experience in our WebRTC calls. Does the audio on these calls sound reasonable compared to when running a file through it? I just want to rule out some integration issue. Regardless, in the meantime we are working on improving Lyra's quality and will share it as soon as we have something.

pyu1538 commented 3 years ago

My subjective listening is good, the audio is clear and sound good. Is the Lyra algorithm not friendly with the PESQ algorithm or kind of this issue?

aluebs commented 3 years ago

That is a great point! Generative models tend to perform poorly with objective metrics like PESQ, POLQA or VISQOL, even if perceptually they sound good, because they aren't optimized to maintain certain alignment or phase. And Lyra is no exception of this rule. Please see section 3.3 and table 1 on our 2017 paper.

aluebs commented 2 years ago

Our listening tests show that the Lyra 1.2.0 release improves quality significantly, but I am guessing that the MOS score would still be low, since it is also based on a generative model.

zxinhui-fb commented 2 years ago

@pyu1538 Hi, Yu, just curious how the Lyra2 runs on a phone call now, since it only supports 3 bitrates , 3200, 6000, 9200. Most of the calls have higher bitrates if opus is used. Thanks.

pyu1538 commented 2 years ago

I do not have the MOS test box now, I will comment here if I figure out a way to measure it.

aluebs commented 2 years ago

I am assuming you have seen our blog post, but just to summarize: From our MUSHRA listening tests, LyraV2@3.2k has similar quality than Opus@10k, LyraV2@6k has similar quality than Opus@13k, and LyraV2@9.2k has similar quality than Opus@14k. But I agree that many calls use Opus at higher bitrates, like 32kbps, and there LyraV2 as it is today won't be able to compete.