I am very interested in experimenting asynchronous pipelining for larger models during my MSc. Specifically, I would like to run T5 with PipeDream-2BW, since it is now supported by Megatron-LM. Is there any way you can help me to update your forked Megatron, like providing a patch with PipeDream-2BW changes or an unsquashed commit history?
Hello Deepak,
I am very interested in experimenting asynchronous pipelining for larger models during my MSc. Specifically, I would like to run T5 with PipeDream-2BW, since it is now supported by Megatron-LM. Is there any way you can help me to update your forked Megatron, like providing a patch with PipeDream-2BW changes or an unsquashed commit history?