search

johannakarras / DreamPose

Official implementation of "DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion"
MIT License
962 stars 73 forks source link

image resize error when fine-tuning UNet #23

Closed weizmann closed 1 year ago

weizmann commented 1 year ago

I got an error when fine-tuning, any updates for the script and instructions? @johannakarras thanks a lot

Originally posted by @weizmann in https://github.com/johannakarras/DreamPose/issues/14#issuecomment-1517602279

weizmann commented 1 year ago

the detailed steps are pasted below:

step 1. run apply_net.py in densepose project to get the dump.pkl (ps: I put the instance images for fine-tuning in /root/workspace/demo/instance_data/)

cd /root/workspace/detectron2/projects/DensePose

python apply_net.py dump configs/densepose_rcnn_R_50_FPN_s1x.yaml \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
/root/workspace/demo/instance_data --output dump.pkl -v

step 2. run the utils/densepose.py to get the *_densepose.npy file in demo/instance_data dir

cd /root/workspace/DreamPose
# change the output_path in utils/densepose.py to /root/workspace/detectron2/projects/DensePose/dump.pkl
python utils/densepose.py

step 3. fine-tuning the UNet with the custom_chkpt(unet_epoch_499.pth) and instance_data_dir(demo/instance_data)

accelerate launch finetune-unet.py --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" --instance_data_dir=demo/instance_data --output_dir=demo/custom-chkpts --resolution=512 --train_batch_size=1 --gradient_accumulation_steps=1 --learning_rate=1e-5 --num_train_epochs=500 --dropout_rate=0.0 --custom_chkpt=checkpoints/unet_epoch_499.pth --revision "ebb811dd71cdc38a204ecbdd6ac5d580f529fd8c"

then I got the error, the detailed stack is pasted below

image

weizmann commented 1 year ago

some kindly recommends: @johannakarras as fine-tuning is crucial for getting a better video result, can you give us a detailed instruction/end-to-end example on how to fine-tune? ☺️

johannakarras commented 1 year ago

I think this issue stems from the transforms.Normalize function. I am able to solve it by removing all "transforms.Normalize([0.5], [0.5])" in the dataset scripts. Please try downloading the latest dataset files and let me know if the issue is not resolved.

weizmann commented 1 year ago

I have used 10 images for the UNet finetune, but I got the cuda out of memory error

image