Is your feature request related to a problem? Please describe.
Is it possible to support model of irregular dtypes? For example, a large multimodal LLM might have a vision encoder that is of dtype=float32 and its LLM part in dtype=bfloat16. This will be particularly helpful since some customized vision models (e.g., MinkowskiEngine) don't support float16/bfloat16.
Describe the solution you'd like
Have a flag (e.g., dont_change_dtype) in DeepSpeedEngine to allow loading a nn.Module model without modifying its dtypes of various parameters (e.g., some params might be float32, while some are bfloat16)
Is your feature request related to a problem? Please describe. Is it possible to support model of irregular
dtype
s? For example, a large multimodal LLM might have a vision encoder that is ofdtype=float32
and its LLM part indtype=bfloat16
. This will be particularly helpful since some customized vision models (e.g., MinkowskiEngine) don't supportfloat16
/bfloat16
.Describe the solution you'd like Have a flag (e.g.,
dont_change_dtype
) inDeepSpeedEngine
to allow loading ann.Module
model without modifying itsdtype
s of various parameters (e.g., some params might befloat32
, while some arebfloat16
)