search

ali-vilab / AnyDoor

Official implementations for paper: Anydoor: zero-shot object-level image customization
https://ali-vilab.github.io/AnyDoor-Page/
MIT License
3.93k stars 359 forks source link

Did you call validation_step during training? #48

Open luccachiang opened 8 months ago

luccachiang commented 8 months ago

Hi, thanks for your great work! I plan to use your model to train on custom dataset. However, when I trained it on a subset of your specified dataset, I did not see a validation procedure. Will the function validation_step be called? Moreover, it seems that you did not have a separate valid_dataloader because only one dataloader was passed into trainer.fit. https://github.com/ali-vilab/AnyDoor/blob/ddcfbafb8fa4f27a2da705a3bcf5bfd2de4fbf98/run_train_anydoor.py#L70 Then how do you save the best model parameters which gave the lowerest loss during validation? Yes, I asked this question because I wanted to save the best model on validation dataset but did not make it yet:( I am not so familiar with pytorch lightning and the way controlnet should be trained, so correct me if I said something wrong:)

Btw, why did you repeat some datasets in the ConcatDataset https://github.com/ali-vilab/AnyDoor/blob/ddcfbafb8fa4f27a2da705a3bcf5bfd2de4fbf98/run_train_anydoor.py#L64C3-L64C3 , but did not repeat image_data?

XavierCHEN34 commented 8 months ago
  1. Validation: No, we did not call validation, instead we infer several training samples to see the converging process.
  2. Picking the best model by observing the lowest loss is not reliable, and neither is calculating some quantitative metrics. So we just save models with different steps.
  3. You could repeat the image_data, it is just a simple way to adjust the ratio of different dataset, feel free to use your own implementation.
ai1361720220000 commented 7 months ago

In the run_train_anydoor.py it seems the training process will not save any model. I'm confused if i should add pl.callbacks.ModelCheckpoint in the callbacks because i'm also not familiar with pytorch lightning..

Dumeowmeow commented 3 months ago

In the run_train_anydoor.py it seems the training process will not save any model. I'm confused if i should add pl.callbacks.ModelCheckpoint in the callbacks because i'm also not familiar with pytorch lightning..

Hi, have you solved it now?I Iran into the same problem and couldn't find where to save the model after training.