search

junyanz / pytorch-CycleGAN-and-pix2pix

Image-to-Image Translation in PyTorch
Other
23.09k stars 6.32k forks source link

Training Freezes #1419

Open aakash-chakraborty1995 opened 2 years ago

aakash-chakraborty1995 commented 2 years ago

Every time I run pix2pix model with facades dataset my training freezes after 300 iterations of 195 epoch. The total number of epoch is 200 and the gpu memory is available too.

How to complete the training process?

junyanz commented 2 years ago

Haven't seen this before. Does --continue_train work for you? You can resume the model training from epoch 195.

raylu1314coding commented 2 years ago

I had the same problem. After that I tried --continue_train but it didn't work . Is the problem because of my GPU only 8G? I used 20000 images to train pix2pix , imgsize 512*256(paired img).

xuzhi0413 commented 2 years ago

I have the same problem. Do you solve the problem？

xuzhi0413 commented 2 years ago

I have the same problem. Do you solve the problem？

junyanz commented 2 years ago

It's hard to know. Every case is different. Would it be possible to share with us some console output / debugging output?

junzhin commented 11 months ago

I also have the same problem!