modelscope / FunCodec

FunCodec is a research-oriented toolkit for audio quantization and downstream applications, such as text-to-speech synthesis, music generation et.al.
https://funcodec.github.io/
MIT License
371 stars 30 forks source link

TKR? #32

Closed a897456 closed 8 months ago

a897456 commented 8 months ago

image The first row in this table, 400 200.... 50 TKR, I think is the sampling rate divided by the stride, and then multiplied by the token, right? just like: 16000/320*8=400TKR, I guess that the number of tokens in each of the first four rows in the table is the same, and they are [8,4,2,1] and the same sampling rate 16000. But, in the last two lines: I wonder that you get the same TKR by changing the sampling rate? Or, by changing tokens?

a897456 commented 8 months ago

image image image From the first two tables, extrapolating to the third table, is there an error, why should the scores of 4.29 and 3.86, in the time domain of M1, not be in the frequency domain of M2 M3? Because I think the first two tables are frequency domain scores, so the third table with the same score should also be frequency domain.

lbehringer commented 8 months ago

Hi, I had similar questions as you.

The TKR is explained here: https://github.com/alibaba-damo-academy/FunCodec/issues/27 The models labeled "FunCodec"/"M1" are actually time-domain models and similar to Encodec (the differences are explained here: https://github.com/alibaba-damo-academy/FunCodec/issues/5). As far as I understand, only the models labeled as "Mag,Angle"/"Mag,Phase" correspond to the FreqCodec (i.e. frequency domain) architecture.

a897456 commented 8 months ago

Hi, I had similar questions as you.

The TKR is explained here: #27 The models labeled "FunCodec"/"M1" are actually time-domain models and similar to Encodec (the differences are explained here: #5). As far as I understand, only the models labeled as "Mag,Angle"/"Mag,Phase" correspond to the FreqCodec (i.e. frequency domain) architecture.

Hi @lbehringer Thank you for your reply to help me understand this article As far as I know:

  1. the tag in Figure 2 in the article: "The overall architecture of the FunCodec models." should be changed to ".... of the FreqCodec models"
  2. This article proposes a new model in the time domain, Funcodec (with some modifications on soundstream and Encodec), and then a new model in the frequency domain, FreqCodec (with some modifications on FunCodec, such as adding domain transformations), right?
ZhihaoDU commented 8 months ago

Hi, I had similar questions as you.

The TKR is explained here: #27 The models labeled "FunCodec"/"M1" are actually time-domain models and similar to Encodec (the differences are explained here: #5). As far as I understand, only the models labeled as "Mag,Angle"/"Mag,Phase" correspond to the FreqCodec (i.e. frequency domain) architecture.

Thanks for your helpful response.

ZhihaoDU commented 8 months ago

Hi, I had similar questions as you. The TKR is explained here: #27 The models labeled "FunCodec"/"M1" are actually time-domain models and similar to Encodec (the differences are explained here: #5). As far as I understand, only the models labeled as "Mag,Angle"/"Mag,Phase" correspond to the FreqCodec (i.e. frequency domain) architecture.

Hi @lbehringer Thank you for your reply to help me understand this article As far as I know:

  1. the tag in Figure 2 in the article: "The overall architecture of the FunCodec models." should be changed to ".... of the FreqCodec models"
  2. This article proposes a new model in the time domain, Funcodec (with some modifications on soundstream and Encodec), and then a new model in the frequency domain, FreqCodec (with some modifications on FunCodec, such as adding domain transformations), right?

The response of @lbehringer is really helpful, and your understanding is almost right. The Figure 2 is used to offer a universal framework of both time-domain and frequency-domain models in FunCodec, therefore the title is "The overall architecture of the FunCodec models."

For item 2, your understanding is right.