layerwise with SGD

[AminSuzani]

Hi,

I use layerwise pre-training as below:

e = theanets.Experiment(theanets.feedforward.Regressor,
                                                layers=(options.featuresN, 200, 300, 200, options.landmarksN*2),
                                                optimize= ['layerwise','sgd'] ,
                                                activation='relu',
                                                )

My code used to work fine with previous versions of the software. However, when I installed a newer version, it seems that the default trainer for layerwise is changed from SGD to NAG. It gives me J=nan. Is there a way that I can use the layerwise pre-trainer with SGD again? or if it's not possible, can you tell me how I can download and install the older version (that used SGD) using git clone?

Thanks, Amin

[lmjohns3]

Hi Amin -

Yes, you can do this, but you'll need to change the way you set up your experiment just a tiny bit, by calling the add_trainer method directly:

e = theanets.Experiment(
    theanets.feedforward.Regressor,
    layers=(options.featuresN, 200, 300, 200, options.landmarksN*2),
    activation='relu',
)
e.add_trainer('layerwise', theanets.trainer.SGD)
e.add_trainer('sgd')
# ...
e.run(...)

Notice also that I've omitted the optimize keyword argument when constructing the Experiment object.

I've tried this with the mnist-autoencoder example, but please let me know if this doesn't work for your project.

[lmjohns3]

Also, I'm curious why you're getting nans when training with the NAG method. Have you tried making the learning rate very small and seeing whether that eliminates the problem? If the nans persist with the NAG trainer, it might be a bug in that trainer -- but if that's the case, please file another issue.

[AminSuzani]

Thanks for your response. Sure, I will play around with the learning_rate and will open another issue if the problem persists. For now, I still have problems with the changes in the latest version, I just opened another issue for it.

[lmjohns3]

Just to check, did the above work for you, so that the layerwise trainer uses SGD instead of NAG? If so, please close this issue and we'll look at the performance in general in the new issue. Thanks!

[AminSuzani]

Yes, it worked. Using "add_trainer", I am now able to to use SGD for layerwise training.