Extensions to relative positional encoding - Githubissues

entity-neural-network / incubator

Collection of in-progress libraries for entity neural networks.

Apache License 2.0

29 stars 10 forks source link

Extensions to relative positional encoding #203

Closed cswinter closed 2 years ago

cswinter commented 2 years ago

Adds several new features to relative positional encoding that allow it to work better in continuous spaces:

adding a linear projection of the relative position/distance to the relative keys/queries
partition functions for relative positional encoding based on the relative angle, distance, or polar coordinates
interpolating between partitions
(partially implemented) rotating relative positions by the orientation of each entity

Other changes not directly related to relpos encoding:

improved xprun builds
new "webpdb" feature to catch crashes and allow for debugging with webpdb
misc bug fixes