search

linhaowei1 / CLoG

✌ CLoG: Benchmarking Continual Learning of Image Generation Models
MIT License
14 stars 3 forks source link

The testing procedure related to C-LoRA. #4

Open Programmergg opened 3 months ago

Programmergg commented 3 months ago

Excellent work! However, I have some concerns regarding the testing results of C-LoRA method. In the sample function of lora_ddim.py, you obtain the specific task index by executing task_id = (labels[0] // self.data_args.class_num).item() and then load the corresponding PEFT-derived parameters based on this task ID. By doing so, isn’t this essentially an ensemble scenario? The only difference is that the model is fine-tuned using PEFT rather than full fine-tuning, which might explain why the C-LoRA results are closer to those of an ensemble. Drawing an analogy to the LoRA-based continual classification scenario, during the testing phase, we cannot directly access task identifiers. Instead, we need to obtain pseudo-task identifiers through some means or directly use the results generated by all LoRA parameters.

Programmergg commented 2 months ago

Additionally, I have a small suggestion regarding the testing process. Currently, you are using normally distributed class labels to evaluate the expected FID on the current task. However, this approach might not fully capture the class-guided generative capability. Therefore, why not generate a certain number of samples for each class directly and calculate the expected FID based on these samples?

linhaowei1 commented 2 months ago

Thanks for your comment! The code for C-LoRA is incorrect. The suggestion for testing FID makes sense and we'd like to try it. We will improve this work in September (add more benchmarks and baselines). And we welcome PRs if you'd like to.

Programmergg commented 2 months ago

Hello, I am very interested in this direction and thrilled to have encountered like-minded individuals. As such, I have been making some optimizations to your code and have started a new fork. If you are interested, feel free to take a look. I will continue making modifications to achieve a more general version. If you don't mind, I will submit a pull request (PR) once it's ready. Thank you again for your valuable contribution.

linhaowei1 commented 2 months ago

Hello, I am very interested in this direction and thrilled to have encountered like-minded individuals. As such, I have been making some optimizations to your code and have started a new fork. If you are interested, feel free to take a look. I will continue making modifications to achieve a more general version. If you don't mind, I will submit a pull request (PR) once it's ready. Thank you again for your valuable contribution.

Cool! I will take a look. We're very excited that you're interested in our work.