Open dxoigmn opened 1 year ago
I should note that I'm not sure this works in multi-gpu mode.
I should note that I'm not sure this works in multi-gpu mode.
This does work but one must beware that BatchNorm
modules get turned into SyncBatchNorm
when using DDP:
https://github.com/IntelLabs/MART/blob/ed89c722f8602885f738cd2765af3d3de97c10af/mart/configs/trainer/ddp.yaml#L8
What does this PR do?
I think it's worth thinking about whether we should merge this PR. Note that
AttackInEvalMode
will be wrong in newer versions of PL since it puts the model intotrain
mode afteron_train_start
is called: https://lightning.ai/docs/pytorch/stable/common/lightning_module.html#hooks.That said, I do not like using
eval
mode since many modules branch onself.training
. I think a better option is to replace batch norm layers with frozen batch norm and remove dropout layers since that is the semantics one actually wants.eval
mode is just abused to do that.Type of change
Please check all relevant options.
Testing
Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.
Before submitting
pre-commit run -a
command without errorsDid you have fun?
Make sure you had fun coding 🙃