Open tuttyfrutyee opened 2 years ago
Hi, Thanks for the question! The model description in the appendix refers to a different model that runs diffusion on the simplex, the main model which runs diffusion on the embedding space just add the time_embeddings directly to input features.
I will release the simplex model & code too in another branch in the future!
I see, thanks for the answer.
Did you guys try to inject the time_embeddings to the different layers of the transformer network, similar to what is done in the vision area for diffusion models?
Hi,
The paper, Appendix F, 4'th bullet point states that time embeddings are incorporated by softmax applied on learnable scaling and offsetting operations. I could not find the part of the code that does this. My first impressions are that time embedding is achieved at 902'th line of the "transformer_model2.py" file, in which the learnable time_embeddings are just linearly added to the input features. Am I missing something? Thank you...