What does this PR do?

Numeric instability if almost inevitable. So let's loosen the enforcer.

Type of change

Please check all relevant options.

[x] Improvement (non-breaking)
[ ] Bug fix (non-breaking)
[ ] New feature (non-breaking)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

[x] pytest
[ ] CUDA_VISIBLE_DEVICES=0 python -m mart experiment=CIFAR10_CNN_Adv trainer=gpu trainer.precision=16 reports 70% (21 sec/epoch).
[ ] CUDA_VISIBLE_DEVICES=0,1 python -m mart experiment=CIFAR10_CNN_Adv trainer=ddp trainer.precision=16 trainer.devices=2 model.optimizer.lr=0.2 trainer.max_steps=2925 datamodule.ims_per_batch=256 datamodule.world_size=2 reports 70% (14 sec/epoch).

[x] The title is self-explanatory and the description concisely explains the PR
[x] My PR does only one thing, instead of bundling different changes together
[ ] I list all the breaking changes introduced by this pull request
[ ] I have commented my code
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I have run pre-commit hooks with pre-commit run -a command without errors

Make sure you had fun coding 🙃