Positional encoding frequency bands should be linearly spaced

lucidrains / perceiver-pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

MIT License

1.1k stars 134 forks source link

Positional encoding frequency bands should be linearly spaced #52

Closed djl11 closed 3 years ago

djl11 commented 3 years ago

A small bug, but as alluded to in this comment by @marcdumon, it seems as though the frequency bands are indeed spaced linearly in the official JAX implementation.

lucidrains commented 3 years ago

@djl11 thanks Daniel, i think you are correct :pray: let me know if this version looks good to you https://github.com/lucidrains/perceiver-pytorch/releases/tag/0.7.4

djl11 commented 3 years ago

Looks good!