Potential experiments info GLOBAL

[shyammarjit]

• [ ] Which linear layer (attention matrices and/or FFN/MLP layers) we would be recommending to finetune?
  1. • [x] We need to identify which weight matrices have the highest importance for the finetuning.
  2. • [ ] In attention matrices, what attention matrices do we want to finetune?
• [x] How does MLP play a role in finetuning?
• [x] What is the optimal value of hyper-parameters?
  1. • [x] Rank for weights matrices (Rank vs. Complexity)
  2. • [x] Learning rate (Reln of LR with Rank and Complexity)
  3. • [x] Iteration
• [ ] How training the text encoder is effective in terms of different prompts?
  1. • [x] In the text encoder, we have KQVO and MLP layers.
  2. • [x] What is its effect on results on different simple-to-complex prompts?
• [ ] How to define the complexity of a text prompt as well as images
• [ ] What’s the effect of inference hyper-parameters such as guidance score and no of steps?
• [ ] What will be the evaluation matrices we can consider?
  1. • [x] May need to read the state-of-the-art papers
  2. • [ ] using DINO/CLIP
  3. • [x] Read https://github.com/zhangjiewu/awesome-t2i-eval
• [ ] What’s the effect of the number of training images?
• [ ] Few-shot (one-shot) settings {How do the results change when using different numbers of images?}
  1. • [ ] Optimal no of training images to obtain full orientation
  2. • [ ] If we have fewer images (let's say just one image) can it able to construct a new orientation from the available orientation? For example, let's say I have only shown only left images, can it construct a new posture of it?
• [ ] How efficiently we can adopt LoRA?
  1. • [ ] GPU memory
  2. • [x] inference time
  3. • [ ] weight storage
  4. • [x] Number of parameters
• [ ] How does chaining the backbone diffuse model help in higher-resolution image synthesis? (SDXL and without SDXL i.e. base
• [ ] How to initialize the lora weight matrics (where we can very this one)
  1. • [x] Random gaussian distribution
  2. • [x] Xavier initialization
  3. • [ ] He initialization
  4. • [ ] Xavier Glorot Initialization

[shyammarjit]

@HarshitSingh-14 Please try to obtain the correlation of KQVO weight matrices and verify once the order of KQVO according to their role and importance. @nityanandmathur you may also take this part from @HarshitSingh-14. Please contact me if you did not get the points behind it. This task is being assigned to both of you @HarshitSingh-14 and @nityanandmathur.