search

openhackathons-org / gpubootcamp

This repository consists for gpu bootcamp material for HPC and AI
Apache License 2.0
513 stars 254 forks source link

[AI][Megtron] Train GPT models with NVIDIA Megatron-LM #49

Open mozhgan-kch opened 3 years ago

mozhgan-kch commented 3 years ago

This Bootcamp is designed to give NLP researchers an overview of the fundamentals of NVIDIA Megatron-LM ( NVIDIA open-source framework for training very large language models). The focus will be on training GPT Megatron models specifically.

It will consist of intro to Megatron-LM code base, converting data to mmap format, understanding model parallel, data-parallel and how to config your training, then train and profile GPT Megatron models.

Zenodia commented 3 years ago

I forked it and created a branch called megatron , i am now ready to create a PR , but i am not sure how you want me to do this. the ready repo can be found here https://github.com/Zenodia/gpubootcamp/tree/megatron

mozhgan-kch commented 2 years ago

Add a link in the start notebook to our other materials. Check the AI materials for reference. Assigned to @mozhgan-kch