Closed manmeet3591 closed 2 months ago
score_deterministic supports multiple GPU using torch.distributed.
The script should work pretty much out of the box if run the script with mpi
or torchrun
and initialize torch distributed before beginning scoring. e.g.
torch.distributed.init_process_group()
# the rest of the script
For parallel use, I might recommend using https://github.com/NVIDIA/earth2mip/blob/86b11fe4ba2f19641802112e8b0ba6b962123130/earth2mip/inference_medium_range.py#L254 instead. This will save one csv file per rank with separate scores for (initial_time, lead_time). Unlike score_deterministic, this approach does not time average these scores.
Closing since the feature already exists.
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Critical (currently preventing usage)
Please provide a clear description of problem you would like to solve.
How to run earth2mip on multi GPU A100, i.e. I want to do distributed training. What changes would be required in