search

minhnhat93 / lfa_sc

Matlab implementation of the paper "Learning fast approximations of sparse coding"
32 stars 22 forks source link

The value of learning_rate.momentum #1

Open huangzehao opened 9 years ago

huangzehao commented 9 years ago

Hi, minhnhat93. I am confused about the value of learning_rate.momentum in your code lista_train. Would you mind telling me how to choose the value? Thank you very much!

minhnhat93 commented 9 years ago

Hi, are you working with Homa? The code on github is very out-dated. The current code is also a big mess. I'm working on it now. The learning_rate.momentum I believe is "to adds a fraction m of the previous weight update to the current one". Value for learning_rate.momentum should be 0.1 to 0.9. My English is bad but you can search gradient descent with momentum on the internet. For example this is a good read: http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/. This learning_rate.momentum in the code is correspond to γ in the article.

Minh Nhat Nguyen Student ID: 51102403 Department of Computer Science and Engineering Ho Chi Minh University of Technology (Vietnam National University) Vietnam Mobile: (84) 966-739-841 Email: minhnhat93@gmail.com

On Fri, Oct 23, 2015 at 4:39 PM, FlyingKid notifications@github.com wrote:

Hi, minhnhat93. I am confused about the value of learning_rate.momentum in your code lista_train. Would you mind telling me how to choose the value? Thank you very much!

— Reply to this email directly or view it on GitHub https://github.com/minhnhat93/lfa_sc/issues/1.

huangzehao commented 9 years ago

Thank you very much. Your answer help me! And I am not working with Homa. I just want to do an implement of this paper "Deep Networks for Image Super-Resolution with Sparse Prior" in iccv 2015. It's based on LISTA and the author didn't share his training code. So I am trying to implement LISTA firstly. I have readed your code and fixed some simple bug. I test it but I found that "lista_test_script.m" takes so much time. Is this test function work?

You are still working this code now? I think it can be simpply implement by some deep learning frameworks,such as caffe, cuda-convnet and so on. Do you have any idea about this?

I am a Chinese student, my English is also bad. But it's nice to communicate with you, a helpful guy, thank you !

minhnhat93 commented 9 years ago

Hi, You are correct. LISTA is just a recurrent neural network unfolded. When I first started with this project I didn't know about much about those deep learning frameworks and the algorithm in the paper were clear therefore I just went ahead and implemented it from scratch. You can use other frameworks to train this I think. The activation function in this LISTA is the soft threshold function is pretty uncommon thus I dont know if it's available in those frameworks. If you can embed the soft thresholding function in those frameworks, LISTA just become a normal feed-forward network with shared parameters, which mean the only difference is you have to take sum of gradients for those parameters, everything else is the same. That lista_test_script I wrote for so long therefore I not totally sure if it's working or not. I'm doing my thesis now so I dont have much time to spent on developing the code anymore except documenting but I will take a look. On the first thought I think that it is working but you have to use a machine with NVIDIA CPU and the code is making it pause after running each test image so you may want to check that too. Best

Minh Nhat Nguyen Student ID: 51102403 Department of Computer Science and Engineering Ho Chi Minh University of Technology (Vietnam National University) Vietnam Mobile: (84) 966-739-841 Email: minhnhat93@gmail.com

On Fri, Oct 23, 2015 at 10:24 PM, FlyingKid notifications@github.com wrote:

Thank you very much. Your answer help me! And I am not working with Homa. I just want to do an implement of this paper "Deep Networks for Image Super-Resolution with Sparse Prior" in iccv 2015. It's based on LISTA and the author didn't share his training code. So I am trying to implement LISTA firstly. I have readed your code and fixed some simple bug. I test it but I found that "lista_test_script.m" takes so much time. Is this test function work?

You are still working this code now? I think it can be simpply implement by some deep learning frameworks,such as caffe, cuda-convnet and so on. Do you have any idea about this?

I am a Chinese student, my English is also bad. But it's nice to communicate with you, a helpful guy, thank you !

— Reply to this email directly or view it on GitHub https://github.com/minhnhat93/lfa_sc/issues/1#issuecomment-150607133.