Ongoing Paper Summaries

[omarsar]

In this issue, I will maintain a few ML and NLP research papers that I have identified to be worth summarizing. Read more about this initiative here #23.

If you are interested in working together on any of the papers to be published on dair.ai, leave your comment below and we will open a new issue where each task will be tracked individually. Feel free to open the issue yourself and link it back to this issue.

If you have any further questions about the process or this project look at our Wiki or join our Slack group to ask for help (I recommend asking in the #help channel).

Papers that have already been assigned will be tagged with the GitHub username of the author.

Feel free to add your suggestions in the comment section below and we can add the paper to the list.

Papers to summarize

• [x] Recipes for building an open-domain chatbot(@cfierro94)
• [ ] Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
• [ ] Experience Grounds Language (@omarsar)
• [ ] DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• [ ] Weight Poisoning Attacks on Pre-trained Models
• [ ] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• [ ] Byte Pair Encoding is Suboptimal for Language Model Pretraining(assigned to @skeshaw)
• [ ] Evaluating Machines by their Real-World Language Use
• [ ] XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• [ ] Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned [assigned to @manisnesan]
• [ ] ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS (assigned to @ngc436)
• [ ] Revealing the Dark Secrets of BERT
• [ ] Reformer: The Efficient Transformer (assigned to @AntonioLopardo)
• [ ] REALM: Retrieval-Augmented Language Model Pre-Training
• [ ] ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (@laibamehnaz)
• [ ] A Primer in BERTology: What we know about how BERT works (assigned to @vsistla)
• [ ] Towards a Human-like Open-Domain Chatbot
• [ ] Neural Approaches to Conversational AI
• [ ] Neural Transfer Learning for Natural Language Processing
• [ ] The Role of Cooperation in Responsible AI Development
• [ ] Scaling Laws for Neural Language Models
• [ ] Language Models are Unsupervised Multitask Learners - a.k.a GPT-2
• [ ] Advances and Open Problems in Federated Learning
• [ ] On the Measure of Intelligence
• [ ] Single Headed Attention RNN: Stop Thinking With Your Head
• [ ] Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset
• [ ] The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• [ ] Deep Learning Based Text Classification: A Comprehensive Review
• [ ] The Problem with Metrics is a Fundamental Problem for AI
• [ ] A Simple Framework for Contrastive Learning of Visual Representations
• [ ] Growing Neural Cellular Automata
• [ ] Attention is not Explanation
• [ ] Deep Learning for Symbolic Mathematics
• [ ] Self-Distillation Amplifies Regularization in Hilbert Space
• [ ] Distilling the Knowledge in a Neural Network

[manisnesan]

@omarsar I started reading this paper[1] & summarized it here https://forums.fast.ai/t/question-answering-system-for-covid-19/68374 . Would appreciate review and feedback.

[1] Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned

[manisnesan]

@omarsar Will convert the above summary into google doc for review & feedback

[omarsar]

@manisnesan looking forward to it!

[omarsar]

Group paper presentation task for the EXTREME paper #72

[constanzafierro]

Hey @omarsar I’d be interested in doing the summary for “Attention is all you need”

[omarsar]

@cfierro94 This will be great! Let me know if you have any questions or need feedback/help with the process.

[AntonioLopardo]

I would like to work on BART

[ngc436]

I would like to take ELECTRA paper

[omarsar]

@JessikaSmith that would be great! Are you on our slack channel group? If not, I can send you an invite.

[ngc436]

@omarsar Thanks, I am already on the slack channel

[Viktor2k]

Started to summarize Longformer!

[omarsar]

I am working on the "Experience Grounds Knowledge" paper.

[laibamehnaz]

Hi @omarsar, I will be working on ERNIE 2.0 :)

[abhirup-dev]

@omarsar I would like to write about Knowledge Distillation

[omarsar]

@maverickdas thanks for your interest to contribute here. Are you thinking of a paper summary or just a survey of the works?

[skeshaw]

Hi @omarsar, I would like to take up the paper "Byte Pair Encoding is Suboptimal for Language Model Pretraining."

[omarsar]

@skeshaw Absolutely! That would be a nice paper summary. Let me know if you need any help getting started. :)

[abhirup-dev]

@maverickdas thanks for your interest to contribute here. Are you thinking of a paper summary or just a survey of the works?

Sorry for the late reply @omarsar. I was thinking of a survey of the works.

[constanzafierro]

Hey there, I'm going to start working on the " BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" summary!

[Viktor2k]

@omarsar Done with Longformer, please sign me up on What Does BERT Look At?

[Viktor2k]

@omarsar Done with that too :)

[omarsar]

@Viktor2k great work! I fixed a few typos and published it. Thanks!

[constanzafierro]

I’m currently working on “recipes for building an open-domain chatbot”

[reemalyami]

Hi Omar

I would like to write about "Ethical considerations in NLP"

[omarsar]

@reemalyami Thank you! Let me know if you have any ideas in mind.

[srichakradhar]

Hi @omarsar I found this ACL paper interesting. I also have read the paper and made a presentation summary: https://drive.google.com/file/d/15QACtsjCiBik6uyuHFXroVcTWL0EiUue/view?usp=sharing Any review and feedback is appreciated.

[omarsar]

@srichakradhar would you like to work on a paper presentation or paper summary? I will provide you some feedback as soon as I get a chance. Thanks.

[srichakradhar]

Sure @omarsar That'd be awesome! Although I have not presented a paper in a meetup format before, I'd love to do it, with your inputs. :)

[tshrjn]

I'd like to work on REALM.

[omarsar]

@tshrjn That's great. Let me know if you need any help.

[AntonioLopardo]

I would enjoy working on the Reformer Summary

[vsistla]

@omarsar - would like to work on "A Primer in BERTology: What we know about how BERT works" - https://arxiv.org/abs/2002.12327

Let me know if someone else is already working on it. I can partner with them if that is the case.

[omarsar]

@AntonioLopardo , @vsistla awesome! Those are both great papers worth summarizing. Let us know if you need any help. :)

[omarsar]

I would enjoy working on the Reformer Summary

@AntonioLopardo please open a separate issue for this and link it back here. Thanks

[omarsar]

@omarsar - would like to work on "A Primer in BERTology: What we know about how BERT works" - https://arxiv.org/abs/2002.12327

Let me know if someone else is already working on it. I can partner with them if that is the case.

@vsistla please open a separate issue for this and link it back here. Thanks

[abhinavkashyap]

@omarsar If no one is summarising Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. I would like to summarise this paper.

[omarsar]

@omarsar If no one is summarising Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. I would like to summarise this paper

@abhinavkashyap please open an issue in this repository and link back to this issue using #27. Here is an example of how you would do this. We will assign you an editor as well to help you in the process (nothing too formal but it helps to have someone that can guide you in the process). You can also join our Slack channel for any questions you may have. Thanks.

[mmiakashs]

Thanks for this initiative, I would love to write summaries for the following three papers (incrementally). If anyone is working on these papers I would love to collaborate. Let me know

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Attention is not Explanation

[skeshaw]

@mmiakashs Hi! You can open a separate issue for each paper summary that you are working on and link it to this repo. The process is outlined in the comment just above yours by Elvis. Let us know if you need any help.

[omarsar]

Thanks for this initiative, I would love to write summaries for the following three papers (incrementally). If anyone is working on these papers I would love to collaborate. Let me know

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Attention is not Explanation

@mmiakashs They are all great choices. If I can suggest, maybe start with the earliest paper which is "Distilling the Knowledge in a Neural Network". Then the other two can be done in whatever order you like -- they are relatively recent papers.

Please also note that when you are working on a paper, it would be nice to open an issue and link it back here. There are some examples in this thread. Let us know if you have any further questions.

[omarsar]

Hi folks, going forward I will be suggesting papers worth summarizing directly in this repository: https://github.com/dair-ai/nlp_paper_summaries/issues

Also, there were a lot of you working on paper summaries. Any updates on these? We will be happy to help if you need it. Thanks.