Open panosapos opened 1 year ago
A list of Intrusive Metrics: Useful for conditional synthesis, as they require a pair of input and "reconstructed" signal.
PESQ Complex metric that calculates the quality of a speech signal. It was suggested by ITU-T as the recommended metric in Telecommunication systems (telephone and VoIP), but now it has been outdated and replaced by PSQM. However, it is still widely used in DL (especially in speech enhancement models), as it provides a good measure of speech quality. ++ available as a torchmetric
STOI Captures Speech Intelligibility, not speech quality, meaning the fraction of words/syllables that a user would understand. More useful in quite noisy environments. Available as a torch metric
A list of non-intrusive methods: Useful for Unconditional Audio Generation
Important: Requires at least 25 minutes of audio, in order to get a stable FAD score
How can we objectively evaluate our model?
Some random thoughts below:
Intrusive v.s. non-intrusive metrics In speech generation, we generally have 2 kind of metrics: intrusive and non-intrusive. Intrusive metrics require a reference signal that will be compared with the output. This means that they can only be used if we have a pair of input and a reconstruction-output. Ofc, non-intrusive metrics do not require such a pair, but the issue is that they are not widely used, or if they do, they are the results of some other pretrained complex model. Maybe we could use an intrusive metric during validation, in order to keep track of training progress?
L1 and L2 distances between speech signals actually makes no sense, as they do not clearly indicate speech quality, let alone speech perception.
Literally, in every paper they use another combination of objective metrics, so there is not a very clear optimal selection for this.