search

jcjohnson / neural-style

Torch implementation of neural style algorithm
MIT License
18.31k stars 2.7k forks source link

comparing two optimizer, adam, lbfgs #328

Open Henry-Yan-HS opened 8 years ago

Henry-Yan-HS commented 8 years ago

Any insights on the performance/ pros, cons of two optimizers:

Can someone give a brief explanation of how both works, what are their respective advantages and disadvantages, and how they performed?

Thanks!

darkstorm2150 commented 8 years ago

a short example is that adam could do larger images at the cost of quality, lbfgs is considered better at generating art, but is limited is resolution because of the high GPU memory footprint.

cablecutter commented 7 years ago

Adam can be useful when using multires or similar scripts that slowly step up the output image size. When Adam has an example init-image from lbfgs the aesthetic is more or less maintained. So, you would step up to your max allowable resolution under lbfgs and then continue with Adam beyond that.

linrio commented 7 years ago

@vic8760 I used the adam to calculate the iamge, actually the quality is lower than that with L-BFGS. The quality of transfered image is coarser.

linrio commented 7 years ago

@cablecutter What's the mean of

... When Adam has an example init-image from lbfgs the aesthetic is more or less maintained ... ?

ProGamerGov commented 6 years ago

The Adam optimizer can be greatly improved in Neural-Style by simply changing it's default parameters to more "stable" values according to my research here: https://github.com/jcjohnson/neural-style/wiki/Fine-Tuning-The-Adam-Optimizer

With the improved parameter values, Adam become a lot closer to L-BFGS in terms of quality.

Another trick you can do to improve quality when using multiscale scale resolution, is to use L-BFGS for the lower image size steps (where most of the major changes occur), and then use Adam for the subsequent larger image size steps.

nile649 commented 6 years ago

Hi, I was wondering if anyone could help me with code for replacing lbfgs with Adam. I would be happy because I want to study effects of different losses.