lucidrains / lightweight-gan

Implementation of 'lightweight' GAN, proposed in ICLR 2021, in Pytorch. High resolution image generations that can be trained within a day or two
MIT License
1.63k stars 221 forks source link

ZeroDivisionError: float division by zero #56

Open Dok11 opened 3 years ago

Dok11 commented 3 years ago

Catch error on 472 676 step (batch size 3)

Traceback (most recent call last):
  File "C:\Users\oleg\AppData\Local\Continuum\anaconda3\envs\lgan_p3.8\Scripts\lightweight_gan-script.py", line 33, in <module>
    sys.exit(load_entry_point('lightweight-gan', 'console_scripts', 'lightweight_gan')())
  File "\lightweight_gan\cli.py", line 185, in main
    fire.Fire(train_from_folder)
  File "C:\Users\oleg\AppData\Local\Continuum\anaconda3\envs\lgan_p3.8\lib\site-packages\fire\core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\oleg\AppData\Local\Continuum\anaconda3\envs\lgan_p3.8\lib\site-packages\fire\core.py", line 463, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\oleg\AppData\Local\Continuum\anaconda3\envs\lgan_p3.8\lib\site-packages\fire\core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "\lightweight_gan\cli.py", line 176, in train_from_folder
    run_training(0, 1, model_args, data, load_from, new, num_train_steps, name, seed)
  File "\lightweight_gan\cli.py", line 66, in run_training
    retry_call(model.train, tries=3, exceptions=NanException)
  File "C:\Users\oleg\AppData\Local\Continuum\anaconda3\envs\lgan_p3.8\lib\site-packages\retry\api.py", line 101, in retry_call
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger)
  File "C:\Users\oleg\AppData\Local\Continuum\anaconda3\envs\lgan_p3.8\lib\site-packages\retry\api.py", line 33, in __retry_internal
    return f()
  File "\lightweight_gan\lightweight_gan.py", line 1055, in train
    inv_scale = (1. / self.D_scaler.get_scale()) if self.amp else 1.
ZeroDivisionError: float division by zero
lucidrains commented 3 years ago

@Dok11 Hi Oleg! Put in a fix https://github.com/lucidrains/lightweight-gan/commit/54eb6d9d68a15a4ecb9387feca7220babddc65b1