search

entity-neural-network / incubator

Collection of in-progress libraries for entity neural networks.
Apache License 2.0
29 stars 10 forks source link

Relative position encoding value gate #222

Closed cswinter closed 2 years ago

cswinter commented 2 years ago

Novel formulation of relative positional encoding which adds an element-wise multiplication of the relative positional values with a projection of the embedding of the corresponding entity. Seems to match or exceed performance of per-entity-type position values, while being much more general since it allows relative positional values to be a function of any (learned) feature of an entity rather than just it's type.

Resolves https://github.com/entity-neural-network/incubator/issues/141

cswinter commented 2 years ago

Basic ablation on MultiSnake task: https://wandb.ai/entity-neural-network/enn-ppo/reports/Relative-position-encoding-value-gate--VmlldzoxODU5MjI2