Image Colorization

[duxiangcheng]

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

[yangxy]

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

[duxiangcheng]

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Thanks for your reply. I have another question. When making my own data set, do I still need the txt file corresponding to the image? If necessary, how to generate the corresponding text?

[duxiangcheng]

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

If we need to generate captions, do we use grayscale images to generate captions during training and inference?

[yangxy]

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction. In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

If we need to generate captions, do we use grayscale images to generate captions during training and inference?

You can use BLIP2/CoCa to generate captions. You should use RGB(gray) images to get the captions during training(inference). Make sure to replace or discard the words such as black and white b&w etc. in the generated captions during inference.