search

locuslab / deq

[NeurIPS'19] Deep Equilibrium Models
MIT License
724 stars 80 forks source link

Can I train my custom MLP model with DEQ in tutorial page? #10

Closed san9569 closed 3 years ago

san9569 commented 3 years ago

Hello

First of all, I'm really appreciate for you to share your awesome code. I want to train a MLP model with DEQ that is shown in Chapter 4 of your nice tutorial page. (http://implicit-layers-tutorial.org/deep_equilibrium_models/)

For example, if f_theta is :

f_theta(z, x) = relu(Wz + x + b)

I wonder whether I can train it with the equilibrium model. Thank you

jerrybai1995 commented 3 years ago

Hi @san9569 ,

Yes, ReLU should be fine! In fact, you may want to check this paper: https://arxiv.org/pdf/2006.08591.pdf. They also use ReLU (but parameterized W differently to ensure provable convergence).

When you use your proposed MLP as layer f, it may get unstable. In that case, you can (1) ensure your parameters are initialized to very small values, (2) add weight decay if needed, and (3) add normalizations if needed.

jerrybai1995 commented 3 years ago

I'm closing this issue for now, but do let me know (and feel free to re-open) if you have other questions!

san9569 commented 3 years ago

I'm closing this issue for now, but do let me know (and feel free to re-open) if you have other questions!

Thanks for your kind answer and for recommending the paper!