Add support for using custom `Environments` and `Strategies` :zap:

ashleve / lightning-hydra-template

PyTorch Lightning + Hydra. A very user-friendly template for ML experimentation. ⚡🔥⚡

4.27k stars 654 forks source link

Add support for using custom `Environments` and `Strategies` :zap: #608

Open amorehead opened 1 year ago

amorehead commented 1 year ago

What does this PR do?

Adds the ability for one to employ custom (e.g., Lightning Fabric) Environments (e.g., SlurmEnvironment) and Strategies (e.g., DeepSpeedStrategy) during model training or evaluation.

Before submitting

[x] Did you make sure title is self-explanatory and the description concisely explains the PR?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[x] Did you list all the breaking changes introduced by this pull request?
[x] Did you test your PR locally with pytest command?
[x] Did you run pre-commit hooks with pre-commit run -a command?

Did you have fun?

Make sure you had fun coding 🙃 :zap:

codecov-commenter commented 1 year ago

Codecov Report

Attention: 29 lines in your changes are missing coverage. Please review.

Comparison is base (bddbc24) 83.24% compared to head (27fd562) 78.60%.

Files	Patch %	Lines
src/__init__.py	37.50%	10 Missing :warning:
src/train.py	60.86%	9 Missing :warning:
src/eval.py	50.00%	8 Missing :warning:
src/models/mnist_module.py	50.00%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #608 +/- ## ========================================== - Coverage 83.24% 78.60% -4.65% ========================================== Files 11 12 +1 Lines 376 430 +54 ========================================== + Hits 313 338 +25 - Misses 63 92 +29 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

gil2rok commented 3 months ago

@amorehead Can you explain your rationale behind adding these additional configs? Super curious to know your thought process.

Is it related to PyTorch Lighting's use of a "strategy" defined here: https://lightning.ai/docs/pytorch/stable/extensions/strategy.html?

And some example strategies:

amorehead commented 3 months ago

Hey, @gil2rok. Yes, Lightning's strategy class is what I had in mind here. I wanted to make it possible with this template to use any arbitrary (e.g., advanced) training strategy such as DeepSpeed with only a few lines of code changes.

gil2rok commented 3 months ago

Hey, @gil2rok. Yes, Lightning's strategy class is what I had in mind here. I wanted to make it possible with this template to use any arbitrary (e.g., advanced) training strategy such as DeepSpeed with only a few lines of code changes.

Because this pull request has not been accepted, how did you use these advanced strategies? Did you just manually add them the PyTorch Lightning trainer arguments?