search

microsoft / ADBench

Benchmarking various AD tools.
MIT License
102 stars 38 forks source link

Make LSTM more LSTM-like #143

Open tomjaguarpaw opened 4 years ago

tomjaguarpaw commented 4 years ago

LSTM logsumexp has 2 added to the sum. Do we know why this is? The earliest occurrence I can see is https://github.com/awf/ADBench/commit/d75f713d8a6fa4f8ff3f988692c1a5c76238abe9#diff-cadabfe323e7f3ac5fe22b5de628d2aeR24

awf commented 4 years ago

Yes that is very odd, and doesn't ring a bell. Let's also make this issue track other LSTM oddnesses: diagonal weight matrix. Any more?