refactor Multitask BERT

[oserikov]

difficulty: medium mentor: @dimakarp1996
requirements: python, ML useful links: Multitask BERT

Idea description

Implement at least one of the following tasks: PAL-BERT, CA-MTL, MT-DNN(more is better) within the DeepPavlov Library

Coding Challenge

Reproduce results on the GLUE benchmark, which are described in the articles https://arxiv.org/pdf/2009.09139 https://arxiv.org/abs/1901.11504 https://arxiv.org/abs/1902.02671 at least for 1 paper (more is better) using Pytorch framework

[arya2910]

hello @oserikov... i would like to work on this project under gsoc 2021.

[Mudassaralimosu]

Hey @oserikov & @IgnatovFedor,I have installed DeepPavlov in my laptop,I want to contribute to the project Refactor Multitask BERT. It would be very helpful if I can get in contact with the mentor of this project @IgnatovFedor to know more about this project and it setup

[Rohan-cod]

Venerated Sir, I hope you are safe and in good health in the wake of prevailing COVID-19. My name is Rohan Gupta and I am a 3rd-year Computer Science undergraduate student at Shri Mata Vaishno Devi University. I have been working with Python, Swift, and deep learning for a couple of years now and have in-depth knowledge of it. I look forward to contributing to this idea as part of this year's GSoC. It would be a great assistance if you could suggest how to get started. My Linkedin Profile:- https://www.linkedin.com/in/aman-kumar-6b31b9127/

[oserikov]

Hey! I will post some further details here ~ in the next 24hrs

[Astreak]

Hey @oserikov ,I am interested to work in this project under GSOC 2021.

[thevasudevgupta]

Hello @oserikov, I would like to work on this project under GSoC 2021. Looking forward for the project details from you.

[milindthakur177]

Hello @oserikov, I would like to contribute to this project in GSoC 2021. Is there any prerequisite task that we have to do? or Are there any specific parts that you want to see on the project proposal?

[preeti13456]

Hello, Myself Preeti Sharma. I have previously worked on ml system and worked as a data analyst previously. I also participate bin Kaggle competition and other contest to developed my skills. That's why want to work on this issue if you can guide about how to proceed that would be really helpful.

[oserikov]

Hey @arya2910 @Mudassaralimosu @Rohan-cod @praj000 @vasudevgupta7 @milindthakur177 @preeti13456 @Abhishek-1Bhatt ! Great to see you here. We have our Multitask Bert based on the TF1, while we're now migrating the whole library onto the newer frameworks. To get more familiar with the MT-bert, you could learn our docs, pipelines and the codebase. See also the recent commits on the migration from tf1 to torch

[Kapil-Pathak]

Hi @oserikov, I am working with Pytorch framework for past 3 years. Thanks for the update post

[QuickLearner171998]

Hi @oserikov, I have experience in writing deep learning models in Keras/Tf and PyTorch. For GSoC 2021 I am interested in contributing to this project. At present, I am working on setting the repo on my local. Kindly guide how to proceed further.

Thanks

[Costub]

Hey @oserikov , I would love to contribute to this prokect in GSoC 2021. I have been working in PyTorch for quite some time now

[MRafaydev]

Hey @oserikov, I want to contribute to this project because I am good at Python and AI, also 1 year experience in Tensor Flow, Keras, I would like to contribute to this project in GSOC 2021.

[benihime91]

Hello @oserikov, I would like to work on this project under GSOC 2021. Could you please clarify what next steps we as interested students should take ?

[danielkornev]

All new projects are in PyTorch

[tuhinnn-py]

Hey @oserikov , I would love to work on this project for GSoC 2021.

[aryasoni98]

Hey @oserikov , I would like to work on this project of GSoC 2021 too.

[Tanu911]

Hello @oserikov . My name is Tanu Agarwal. I am a third year undergraduate from IIT Guwahati. I am really interested in working on this project. I have been working on NLP for past whole year and would be extremely glad to contribute to refactor Multitask BERT. My LinkedIn profile: https://www.linkedin.com/in/tanu911/

[oserikov]

@dimakarp1996 , could you please share your expertise on all these great questions?

[oserikov]

We're starting regular review sessions of your application proposals drafts.

Every Thursday you can submit Google doc with your proposal (remember the limit of 3 final proposals in total), enable commenter access in the provided link. You won't have to re-submit your proposal(s) if the link stays the same.

Proposals should follow our released template

Our mentors will review them and provide feedback on a weekly basis.

GForm for proposals review: https://forms.gle/2PoHAgv9rjR1fuug7

[ShalinJain1603]

Hello, @oserikov @dimakarp1996. I am Shalin Jain, a Pre-Final Year Student from IIT Guwahati. I have done several projects in the field of NLP and have been working on it for more than a year. I will be glad to with you on making refactor Multitask BERT a success.

[KevinMathewT]

Hey, @oserikov and @dimakarp1996, I'm Kevin Mathew, a senior year undergraduate from BIT Mesra, with relevant experience in PyTorch and NLP. So, for the coding challenge, I was reading the CA-MTL paper and thought I give the challenge a try. I have not implemented much, just the forward pass, I took help from the official implementation of the paper, do check it out https://colab.research.google.com/drive/1tkcvGGtYyshJGCCxTySnoGy-CSIUwTAq?usp=sharing.

[krutikabapat]

Hey @oserikov. I am Krutika Bapat, a final year MS student in computer science at Arizona State University. I have a good amount of internship experience in NLP and have done the latest Internship at American Express AI Labs where I have exclusively worked on the BERT models for Question Answering. I have used BERT on SQUAD data for transfer learning. I would really like to contribute to this project and collaborate. I have worked on different datasets such as Yelp Reviews, Glue, LibriSpeech. My github profile: https://github.com/krutikabapat NLP Projects on github: https://github.com/krutikabapat/Inverse_Cooking_recipe_Generation_from_food_images Linkedin profile: https://www.linkedin.com/in/krutika-bapat-66bb10136/

[dimakarp1996]

@KevinMathewT Hello. Thanks a lot for your dedication, that's a good start, carry on!

[rimijoker]

Hey @oserikov @dimakarp1996 I read the paper Bert and Pals and I came across the implementation by the authors(this one - Bert-n-Pals) which is implemented in pytorch and uses most of the code from Hugging Face Transformers library. Furthermore, upon going through the multitask_bert.py, I found the code uses the tf1 framework and even deeppavlov.core still seems to be using tf1. So would I need to refactor everything everything in multitask_bert to use the pytorch framework and also change some of the core files to make it work, am I going in the right direction? Also it looks like I can reuse code from the authors implementation, would that be fine?

[keeganNull]

Hey @oserikov , I have some problems when implementing the coding challenges. Because of the GCF, I have some problems when running the code build_model(bert_config, download = True). I'm trying to deal with it. If I can't deal with it, I wonder if it's ok to implement it on the colab?

[dimakarp1996]

@keeganNull Could you provide me please with a traceback of error that comes out while running your code? Also, what are the properties of your GPU device what the CUDA /CUDNN versions are and what model are you trying to build? Maybe you just run out of memory?

[dimakarp1996]

@rimijoker Yes, you need to use the pytorch framework. You can also reuse authors code, its OK.

[keeganNull]

@keeganNull Could you provide me please with a traceback of error that comes out while running your code? Also, what are the properties of your GPU device what the CUDA /CUDNN versions are and what model are you trying to build? Maybe you just run out of memory?

The problem is that I can't install the package "hdt". And it showed that there is something wrong with my VSC. I'm trying to deal with it. Anyway, thank you.

[Astreak]

Hey @oserikov @dimakarp1996 hope you are having a good day... I am using the MT-DNN code provided by the author(Link: https://github.com/namisan/mt-dnn) and trying to integrate the functionalities and scripts in the deeppavlov library.But,I came across few demo datasets(Toy datasets) used to show the training process/GLUE benchmark, should I add the datasets or just focus on the functionalities provided the data structure.If i am doing anything wrong let me know.

[dimakarp1996]

@praj000 I think that adding these toy datasets and using them in the training process is a great idea, as using them can speed up debugging of your code.

[keeganNull]

Hey mentor! I'm trying to implement mt dnn.And there is some problems when I do it. So should I have to implement it totally to get chance to be accepted.?I mean I just use some packages like hugging face, maybe I can't do as good as the paper, even worse I can't finish it.Do it matter a lot?

[oserikov]

hey @keeganNull !btw remember to enable commenter access when submitting your proposal this thursday. We were unable to suggest smth

[keeganNull]

hey @keeganNull !btw remember to enable commenter access when submitting your proposal this thursday. We were unable to suggest smth

I have handled it. Thank you.

[Ahnsun]

Hey,@oserikov @dimakarp1996,I am Yu En, a postgraduate major in ML in HuaZhong University of Science and Technology from China, I am mainly engaged in CV research, and used to learn about transformer and Bert. This is my first time to take part in GSoC.I would really like to work on and contribute this project for GSoC 2021. I will really appreciate it if I can get your suggestion.

[harsh1245-bit]

Hey,@oserikov @dimakarp1996, I am Harsh Pandey, a sophomore major in ML in Indian Institute of technology BHU Varanasi in India, I am mainly engaged in Machine learning, Deep learning, Natural Language Processing, and used to learn about transformer and Bert. I am currently also a data science intern. This is my first time to take part in GSoC. I would really like to work on and contribute this project for GSoC 2021. I will really appreciate it if I can get your suggestion.

[milind-soni]

Hello! @oserikov and team! you are doing really exciting work and I would love to be a part of it throughout the summers. I am a third-year electronics and communication undergraduate from India and have an interest in Natural language processing and speech-related projects. You have some exciting projects that I would like to get my hands on and hopefully learn a lot.

[keeganNull]

Hi ！@dimakarp1996
I really need your feedback before the deadline. Should I submit again?

[dimakarp1996]

Hello @keeganNull. I have written a comment, please check your proppsal out.

[dimakarp1996]

@rimijoker Hello. I kindly remind you to add schedule to your proposal, as I haven't found it yet.

[dimakarp1996]

@Iotaa Are you sure that I am your prospective mentor? From your proposal, as your task is Refactor KBQA , maybe you should write to @dmitrijeuseew?

[lotaa]

@dimakarp1996, I'm very sorry ... Thank you for your response

[rimijoker]

@dimakarp1996 I have added a schedule and a new comment to my proposal, please take a quick look :)

[mikewakawski]

Are you still looking for applicants for this project? If yes, then I'd like to help out on this project under GSoC 2021

[dimakarp1996]

@mikewakawski @milind-soni Yes, we are glad to see new proposals.

[dimakarp1996]

@rimijoker I am perfectly satisfied with your schedule.

[keeganNull]

Hello @keeganNull. I have written a comment, please check your proppsal out.

Thank you for your comment. I have checked the comment. And I have found a new multi-task model. What should it at least two models here? As I think, one model like Unicorn (which I found) maybe is easy to implement. That somehow I think I need more tasks to finish a ten-week schedule.
And I'm still finding new models. I think I will finish my proposal before tomorrow. I wish you can check it again at the time.

[dimakarp1996]

@keeganNull I have added the comment about the additional requirement to the MT models; long story short, it's not only about ease of implementation, but also about computational restrictions. (Model should be able to run on inference at 1 GeForce 1080Ti)

[keeganNull]

@keeganNull I have added the comment about the additional requirement to the MT models; long story short, it's not only about ease of implementation, but also about computational restrictions. (Model should be able to run on inference at 1 GeForce 1080Ti)

I have finished my final proposals. Please check it again.

[dimakarp1996]

@keeganNull I have checked it, please look out my answer.