Open 3DTOPO opened 8 years ago
Here is a case where it was detected and no action taken (I didn't override anything for this output - I only added the logic and a printf statement):
...
Setting up style layer 30 : relu5_1
WARNING: Skipping content loss
Running optimization with L-BFGS
Iteration 1 / 1500
Content 1 loss: 841378.984375
Style 1 loss: 520660.522461
Style 2 loss: 97578859.375000
Style 3 loss: 22274044.921875
Style 4 loss: 572314812.500000
Style 5 loss: 28147.811890
Total loss: 693557904.115601
loss from last iteration: -693557904.115601
<optim.lbfgs> creating recyclable direction/step/history buffers
Iteration 2 / 1500
Content 1 loss: 841378.984375
Style 1 loss: 520660.522461
Style 2 loss: 97578859.375000
Style 3 loss: 22274044.921875
Style 4 loss: 572314812.500000
Style 5 loss: 28147.811890
Total loss: 693557904.115601
********** 0 loss from last iteration **********
<optim.lbfgs> function value changing less than tolX
You can see my added printf statement (2nd to last line).
I submitted a pull #97 request for the issue. It carries on now instead of failing with a tolX error for me under these conditions.
I have noticed that when using noise for the seed sometimes it fails with the message "function value changing less than tolX" and all I have to do is keep rerunning the same command until it runs - then all is fine.
Further more, I have determined that when this happens, the total loss between the first iteration and the second iteration is 0.
I added some logic to catch it
around line 299:
I tried reinitializing the image like this, and it carried on where it would have stopped with the tolX error, but I get a blank image:
I have never used torch before, so I feel a little lost still! Oh yeah, I put the image init stuff in a function so I could reuse it (above):
The whole file is attached.
neural_style.lua.txt