Add multigpu pt - Githubissues

caikit / caikit-nlp

Apache License 2.0

12 stars 49 forks source link

Add multigpu pt #288

Open gkumbhat opened 1 year ago

gkumbhat commented 1 year ago

Changes

Add FSDP configuration for PT trainer
Add torch elastic launch

NOTE: This PR to be merged only after merge and rebase with #287

This currently gives following error:

ValueError: expected to be in states [<TrainingState.IDLE: 1>] but current state is TrainingState.FORWARD_BACKWARD