Closed drxmy closed 1 year ago
I find that microsoft's Megatron-DeepSpeed has such feature(https://github.com/microsoft/Megatron-DeepSpeed/pull/56). It is a relatively new PR. I am not sure whether bigscience merge it or not.
I find that microsoft's Megatron-DeepSpeed has such feature(https://github.com/microsoft/Megatron-DeepSpeed/pull/56). It is a relatively new PR. I am not sure whether bigscience merge it or not.