Transformers in keras? - Githubissues

jeffheaton / t81_558_deep_learning

T81-558: Keras - Applications of Deep Neural Networks @Washington University in St. Louis

https://sites.wustl.edu/jeffheaton/t81-558/

Other

5.71k stars 3.04k forks source link

Transformers in keras? #105

Closed randomgambit closed 2 years ago

randomgambit commented 3 years ago

hi @jeffheaton and thanks for this incredible course. Probably one of the best available online!

I was wondering if you are planning to add some notebooks on Transformers in your NLP chapter of the course. I think Transformers are replacing LSTMs (in particular for text classification) and it would be interesting to see how they work with keras.

What do you think? Thanks!

jeffheaton commented 3 years ago

Yes, that is on my list! Need to look for some good library/support for this, maybe for fall 2021. Transformers are not something I touch on a daily basis.

jeffheaton commented 2 years ago

Just added:

https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_10_4_intro_transformers.ipynb
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_10_5_keras_transformers.ipynb

Will also add BERT to the NLP module next semester.