[paved path] basic charnn with GPT example - Githubissues

facebookresearch / recipes

Recipes are a standard, well supported set of blueprints for machine learning engineers to rapidly train models using the latest research techniques without significant engineering overhead.Specifically, recipes aims to provide- Consistent access to pre-trained SOTA models ready for production- Reference implementations for SOTA research reproducibility, and infrastructure to guarantee correctness, efficiency, and interoperability.

BSD 3-Clause "New" or "Revised" License

284 stars 35 forks source link

[paved path] basic charnn with GPT example #27

Closed hudeven closed 2 years ago

hudeven commented 2 years ago

1) We decide to put paved path examples in torchrecipes repo 2) This is an initial commit based on previous work https://github.com/aivanou/disttraining and https://github.com/dracifer/disttraining 3) It's a basic example without any external dependencies(mlflow logging, airflow, torchx, etc), which will be added soon

Testing:

single GPU python main.py
multi GPUs torchrun --nnodes 1 --nproc_per_node 4 \ --rdzv_backend c10d \ --rdzv_endpoint localhost:29500 \ main.py
torch single GPU torchx run -s local_cwd dist.ddp -j 1x1 --script main.py
torchx multi GPUs torchx run -s local_cwd dist.ddp -j 1x4 --script main.py

facebook-github-bot commented 2 years ago

@hudeven has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

hudeven commented 2 years ago

@edward-io yes, thanks for taking a look! As it's the initial check in from previous prototyping repos, there are a lot of things to improve! Any feedback is welcome!

facebook-github-bot commented 2 years ago

@hudeven has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot commented 2 years ago

@hudeven has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

hudeven commented 2 years ago

@ananthsub thanks for the comments! All of them make sense. I plan to adopt those libs soon. Currently, the main goal of this pull request is to add the basic example and then have pipeline setup config/instructions on top of it.