Killed Error on Training

maum-ai / faceshifter

Unofficial PyTorch Implementation for FaceShifter (https://arxiv.org/abs/1912.13457)

BSD 3-Clause "New" or "Revised" License

610 stars 114 forks source link

Killed Error on Training #5

Closed CengizhanYurdakul closed 3 years ago

CengizhanYurdakul commented 3 years ago

Hello, thanks for sharing codes. I tried to train with batch size 2, 4, 8 and 16 seperately on v100 16GB but i got an error "Validation sanity check: 0it [00:00, ?it/s]Killed". Do you know what is the problem?

usingcolor commented 3 years ago

I think there are some error occurred while validation loop (https://pytorch-lightning.readthedocs.io/en/0.4.9/Trainer/Validation%20loop/, https://github.com/PyTorchLightning/pytorch-lightning/issues/2295). Did you use the docker image? If you give me some details of your environment, then it will be helpful to solve your problem. I recommend you biggest batch size that you can, bigger works better.