Open Tomsawyerhu opened 1 month ago
Thank you for your nice work. It will take some time for me to digest your code. We did some experiments with GPT2 (not published) and noticed that it benefits from explaining the softmax classification output with temperature scaling. Not so important now, just wanted to keep this in mind.
I write a lrp-backward version for gpt2 and provide unit tests. Generally, two functions (conv1d and baddbmm) and one module(Conv1d) are newly added. Also, I modify source code of GPT2, for convenience, dropout layer are ommited.