mbaddar1
commented
6 months ago Sent to martin
@meigel Hi Martin , hope you are doing well :) . I am just touch-basing with you to communicate my progress so far I am attaching a (DRAFT) research proposal for Tensor-Train DDPM models. I have put the layout based on your recommendations. As it is still in the DRAFT phase I am not asking for a review yet from you. The important thing I have added details for is section 4.1 on the architecture for approximating the parametric noise for DDPM using Functional-Tensor-Train (FTT) and two different basis functions : Legendre Polynomials and B-Splines. This is to answer your question about how FTT can model non-stationary processes like DDPM , contrary to the Tensor-Train Density Estimation work. The core equation that would answer this is eq.13 p8 in the draft-proposal. The important section I am working on is section 4.2 on the optimization of the proposed architecture. I am planning to experiment two method : Alternating Linear Scheme and Riemannian Gradient Descent The rationale behind experimenting ALS with DDPM is i) the loss in DDPM is L2 loss to which ALS can be directly applied ii) I had a discussion with Charles about his experiments with David about applying FTT to Rectified-Flows . Charles and I had a long meeting through walking through the code and he has good initial results with 2-cluster Gaussian Mixture with TT-RecFlow. I think this something I can base my work on The rationale behind experimenting with RGD is that it has been successfully applied in TTDE context and other TT-ML context, so as a plan B if ALS failed, RGD would be something worthy to experiment Also I have the details of planned experiments in section 5 , like toy- and real-world datasets to use, quality measure etc.. Next Step As Charles shared the TT-Recflow code with me , I will spent time experimenting with it to i) gain more experience with ALS opt. and ii) get ideas that can be useful in my TT-DDPM work, as already RecFlow has some similarities to DDPM in both the domain and objective function. Based on the experience and idea I will get from step A , I will add the details of section 4.2 (optimization) in the proposal I hope you are having a long weekend and accept my apologies for connecting in your non-working day. Once I have more "significant" progress, I will contact you for a f2f meeting. (edited) PDF
From Martin comments https://uq-berlin.slack.com/archives/D0168AT80RY/p1713873401234819 "If the plan is to use TTs in DDPM (which is absolutely worth a shot), this has to be described (in a 5-10 pages research plan, not an overview document) in full detail but only what is required for the new method. Goals, structure, possible analytical results, algorithms etc. etc."
To be finished on 1 May 2024