search

jcjohnson / neural-style

Torch implementation of neural style algorithm
MIT License
18.31k stars 2.71k forks source link

problem with NANs with vgg16 #423

Open wulabs opened 6 years ago

wulabs commented 6 years ago

I've been getting NAN output with vgg16 ~50% of the time resulting in a black or scrambled-tv-reception output.

With vgg19 and NIN it happens less than 0.1% of the time.

I'm using all the same params other than the default layers and models. I'm using 700px style size, and 50:5 style to content ratio. I'm using CuDNN on GPU with LBFGS correction set to 100.

I have occasionally found that very low style:content ratios (< 5) cause the NANs but i haven't found a consistent threshold to enable 100% non-NAN output.

Does anyone have any tips on how to fix or tweak so I don't get any more NANs for vgg-16? Is there a different vgg-16 model I can try?

ProGamerGov commented 6 years ago

In my tens of thousands of tests, it's only happened to me a few times. Normally I see that the loss values went to extreme values before it happened, indicating that for some reason, things were stable. To fix it, I literally just add +1 or -1 to the image_size value that I was using.