Siamese Multi-depth Transformer-based Hierarchical Encoder

[lalitpagaria]

🌟 New model addition

Model description

Recently Google is published paper titled "Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching". And according to paper for long-form document matching SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT.

I feel it is will add value to already awesome transformers models collection :slightly_smiling_face:

Open source status

• [X] the model implementation is available: https://github.com/google-research/google-research/tree/master/smith
• [X] the model weights are available: SMITH-WP+SP model checkpoint and GWikiMatch data
• [X] who are the authors: https://github.com/yangliuy, https://github.com/eladeban

[lalitpagaria]

Linking Haystack issue https://github.com/deepset-ai/haystack/issues/719

[Engineering-Geek]

Frequent user of hugging face here, I'm a fan of this new publication and would love to see it implemented. Commenting here for the GitHub algorithm to ++

[ChanCheeKean]

Hi all, rather than waiting for the implementation in huggingface. Is there a simple way to utilize the pretrained model from the smith repo on our own dataset (to generate document embedding)?