Closed robertodessi closed 3 years ago
Fixing storing interaction callbacks
When training with multiple gpus interactions are not always aggregated across gpus. Given that we should handle this and control whether the leader process only (aggregated case) or all process (non-aggregated case) should save them.
UTs pass locally
Description
Fixing storing interaction callbacks
Related Issue (if any)
207
Motivation and Context
When training with multiple gpus interactions are not always aggregated across gpus. Given that we should handle this and control whether the leader process only (aggregated case) or all process (non-aggregated case) should save them.
How Has This Been Tested?
UTs pass locally