search

shivammehta25 / Matcha-TTS

[ICASSP 2024] 🍵 Matcha-TTS: A fast TTS architecture with conditional flow matching
https://shivammehta25.github.io/Matcha-TTS/
MIT License
746 stars 95 forks source link

Question about OT-CFM #112

Open umiuri1105 opened 2 weeks ago

umiuri1105 commented 2 weeks ago

Thank you for your great work. I have a question about Flow Matching. The Matcha-TTS paper states that OT-CFM is used, but where is the OT-CFM program implemented in this GitHub program? It looks like the program implements general CFM (I-CFM).

shivammehta25 commented 2 weeks ago

Hello thank you for your interest in our work.

Matcha-TTS, uses the conditional Optimal Transport introduced in Flow Matching for Generative Modeling . In section 4 Example II, the equation of the flow is described in more detail with a disclaimer.

Lastly, we note that although the conditional flow is optimal, this by no means imply that the marginal VF is an optimal transport solution. Nevertheless, we expect the marginal vector field to remain relatively simple.

We use the same definition and formulation.

I am sorry, I am not familiar with I-CFM, could you please provide a reference to it so I can take a look at it?

umiuri1105 commented 2 weeks ago

Thank you for your response. I-CFM is explained in Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport. OT-CFM is also discussed in this paper. When implementing OT-CFM, is it necessary to use the library "pot"?

shivammehta25 commented 2 weeks ago

I-CFM is explained in Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport.

Ah ofcourse I have read this article, I just couldn't recall that they call it I-CFM. I think they have their implementation open sourced. And they seem to be using pot and from what it looks, it just provides a sample plan off the shelf.

RU-1000 commented 2 weeks ago

Hi! I thought the same thing as @umiuri1105 . I could only find CFM too. (https://github.com/shivammehta25/Matcha-TTS/blob/main/matcha/models/components/flow_matching.py#L87) In which file is optimal-transport conditional flow matching (OT-CFM) implemented?

shivammehta25 commented 2 weeks ago

Hello,

What is referred to as OT-CFM here is the conditional OT-CFM introduced in Flow Matching for Generative Modelling, not the marginal one which was later introduced in Tong et al. where they called I-CFM.

umiuri1105 commented 1 week ago

Hello, and I apologize for my delayed response. I have three questions:

  1. Are there actually two different Flow Matching methods called OT-CFM? (One proposed in "Flow Matching for Generative Modelling" and another in "Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport")
  2. What is the difference between CFM and OT-CFM?
  3. Currently, I think that OT-CFM is implemented in the program. If Matcha-TTS were implemented with CFM instead, what would the program look like?
shivammehta25 commented 1 week ago

To my understanding

  1. Yes! But one is conditional while the other is not and forms Mini batched Transport paths before (a significant overhead preventing its use in practical applications) and then trains on those pairs.
  2. CFM is an umbrella framework of which OT-CFM is one way to form these probability flow paths. In "Flow Matching for Generative Modelling" they also proposed a way to formulate CFM as diffusion.
  3. You will just need to modify the computation of the conditional vector field to regress onto: https://github.com/shivammehta25/Matcha-TTS/blob/77804265f877b0c42f13cfdece6541dde7838090/matcha/models/components/flow_matching.py#L112 Currently it uses eq (22) from the paper Lipman et al..