search

allenai / deep_qa

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)
Apache License 2.0
404 stars 132 forks source link

Adding attention layers #150

Closed matt-gardner closed 7 years ago

matt-gardner commented 7 years ago

This PR does two things:

  1. It moves common.tensors to its own top-level module, tensors, and adds some structure to the module.

  2. It adds a tensors.similarity_functions module, which allows for configurable similarity computations, which form the core of an Attention layer. Along with this, it adds a layers.attention module, with a couple of Attention layers, and merges what was layers.softmaxes with it.

matt-gardner commented 7 years ago

Well, the last failure is one that's not so simple. Turns out you can't use K.repeat_elements if you don't know how many times to repeat, because your input shape has a None... I think I know how to fix this, but I'm done for today.