Closed songkq closed 1 year ago
@songkq Thank you. The reason is because we don't set the loading type for these two bias. Fixing in this commit https://github.com/NVIDIA/FasterTransformer/commit/0c128050e14b65d72b3c28c0324cc9db6f677be8.
Solve the problem with https://github.com/NVIDIA/FasterTransformer/issues/561.
@Thytu @rr0gi Hi, could you please give some advice for this issue?
nemo_megatron_mt5_3b_bf16_tp2.nemo
(https://huggingface.co/nvidia/nemo-megatron-mt5-3B) model was trained with--tensor_model_parallel_size=2.
I have converted the
nemo-megatron-mt5-3B
to binary files successfully bypython3 FasterTransformer/examples/pytorch/t5/utils/nemo_t5_ckpt_convert.py -i nemo-megatron-mt5-3B/nemo_megatron_mt5_3b_bf16_tp2.nemo -o ./models/nemo-megatron-mt5-3B/ -m mt5-3B -i_g 2
When run a tritonserver with
CUDA_VISIBLE_DEVICES="0,1" /opt/tritonserver/bin/tritonserver --model-store=fastertransformer_backend/all_models/nemo-megatron-mt5-3B/
, tritonserver failed to loading the model with unmatched shape.Here are my config.pbtxt and config.ini. config.zip