Let's select a paper for our paper reading/discussion session on June 13, 2020

[omarsar]

Comment a paper you would like us to discuss during our weekly paper reading discussion.

You can vote on a suggested paper by using the 👍 emoji. I will close the issue in one or two days and select the paper with the most votes. Then I will make the announcement at the beginning of the week. Thanks.

[RachitBansal]

When BERT Plays the Lottery, All Tickets Are Winning

https://arxiv.org/abs/2005.00561

The first author is even a part of the dair.ai community.

[sandyasm]

The Unstoppable Rise of Computational Linguistics in Deep Learning

[ArmiNouri]

Evaluating NLP Models via Contrast Sets: https://arxiv.org/pdf/2004.02709v1.pdf

[msank00]

Title: Weight Poisoning Attacks on Pre-trained Models

• Paper link
• Code
• ACL2020

Why important: Operates in the emerging area of security in NLP and focuses on attacking a pre-trained language model that is fine-tuned on data of a target task.

Idea:

• It is a proof-of-concept algorithm for poisoning the weights of a pre-trained model (such as BERT, XLNet, etc...) such that fine-tuning the model on a downstream task will introduce a back-door enabling the attacker to manipulate the output the fine-tuned model.

• Provide a poisoned model for others to download that we can then later exploit. In particular, the authors propose weight poisoning attacks that inject vulnerabilities into a model that exposes “backdoors” after the model is fine-tuned on a target task. Trigger words are rare words (arbitrary nouns also work) that are inserted in a sequence and enable switching the model’s prediction to a target label.

[manisnesan]

Title: Learning distributed representation of concepts Author: Geoffrey E. Hinton Link

Twitter Thread which inspired to read this seminal paper from 1986.

[KhalidAlt]

Title: Generating Long Sequences with Sparse Transformers Link

The GPT3 used something similar to sparse transformer and they cited this paper. In addition, we could try to implement the paper as task in the project of ( paper_implementations ).

[eyebies]

BLEURT: Learning Robust Metrics for Text Generation https://ai.googleblog.com/2020/05/evaluating-natural-language-generation.html?m=1 https://youtu.be/rl4nUngiR2k

[omarsar]

End-to-end Object Detection with Transformers

https://ai.facebook.com/research/publications/end-to-end-object-detection-with-transformers