I think in the last line the exp should be divided by min_timescale rather than multiplied, since it's inverse timescales. Usually min_timescale is 1 so it doesn't matter. But e.g. if you fix max_timescale and change min_timescale, the resulting inverse timescale corresponding to max_timescale changes.
A simpler implementation could be roughly something like this:
and from this one you can derive the current implementation, except with division instead of multiplication. It can be even simpler with logspace but tf seems to have this function only as experimental.
Hi,
There might be a small bug here:
https://github.com/tensorflow/tensor2tensor/blob/ef1fccebe8d2c0cf482f41f9d940e2938c816c78/tensor2tensor/layers/common_attention.py#L445-L449
I think in the last line the
expshould be divided bymin_timescalerather than multiplied, since it's inverse timescales. Usuallymin_timescaleis 1 so it doesn't matter. But e.g. if you fixmax_timescaleand changemin_timescale, the resulting inverse timescale corresponding tomax_timescalechanges.A simpler implementation could be roughly something like this:
and from this one you can derive the current implementation, except with division instead of multiplication. It can be even simpler with logspace but tf seems to have this function only as experimental.
Let me know if this makes sense.
Thanks a lot!