where the checkpoint can be reshaped for a different TP/PP target for Megatron-Deepspeed to Megatron-LM, we need the same for Megatron-Deepspeed to Megatron-Deepspeed. i.e. currently it is not possible to change the TP topology once the training started.
Feature request
Similar to https://github.com/microsoft/Megatron-DeepSpeed/tree/main/tools/convert_checkpoint
where the checkpoint can be reshaped for a different TP/PP target for Megatron-Deepspeed to Megatron-LM, we need the same for Megatron-Deepspeed to Megatron-Deepspeed. i.e. currently it is not possible to change the TP topology once the training started.
So the desired API is:
Critical new need: the optimizer states need to be reshaped as well
Thank you!
@tjruwase