search

triton-inference-server / fastertransformer_backend

BSD 3-Clause "New" or "Revised" License
411 stars 133 forks source link

model output must specify 'data_type' for fastertransformer #119

Open 18810251126 opened 1 year ago

18810251126 commented 1 year ago

Description

按照教程跑CUDA_VISIBLE_DEVICES=1,2 mpirun -n 1 --allow-run-as-root /opt/tritonserver/bin/tritonserver  --model-repository=${WORKSPACE}/all_models/bert/ 命令时报错：
E0412 06:53:22.368746 15182 model_repository_manager.cc:1927] Poll failed for model directory 'fastertransformer': model output must specify 'data_type' for fastertransformer

Reproduced Steps

CUDA_VISIBLE_DEVICES=1 mpirun -n 1 --allow-run-as-root /opt/tritonserver/bin/tritonserver  --model-repository=${WORKSPACE}/all_models/bert/
byshiue commented 1 year ago

Please provide your reproduced steps.

18810251126 commented 1 year ago

docker run -it --rm --gpus=all --shm-size=1g --ulimit memlock=-1 -v ${WORKSPACE}:${WORKSPACE} -w ${WORKSPACE} ${TRITON_DOCKER_IMAGE} bash

now in docker

export WORKSPACE=$(pwd)

sudo apt-get install git-lfs git lfs install git lfs clone https://huggingface.co/bert-base-uncased # Download model from huggingface git clone https://github.com/NVIDIA/FasterTransformer.git # To convert checkpoint export PYTHONPATH=${WORKSPACE}/FasterTransformer:${PYTHONPATH} python3 FasterTransformer/examples/pytorch/bert/utils/huggingface_bert_convert.py \ -in_file bert-base-uncased/ \ -saved_dir ${WORKSPACE}/all_models/bert/fastertransformer/1/ \ -infer_tensor_para_size 2

sudo pip3 install torch==1.12.1+cu116 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html /workspace/build/fastertransformer_backend/build/bin/bert_gemm 32 32 12 64 1 0 2

cp config.pbtxt ${WORKSPACE}/all_models/bert/fastertransformer/ CUDA_VISIBLE_DEVICES=0,1 mpirun -n 1 --allow-run-as-root /opt/tritonserver/bin/tritonserver --model-repository=${WORKSPACE}/all_models/bert/ &

Reference link: https://github.com/triton-inference-server/fastertransformer_backend/blob/main/docs/bert_guide.md

byshiue commented 1 year ago

What's the version of your docker image?

18810251126 commented 1 year ago

TRITON_VERSION=22.03， Docker version 20.10.17, build 100c701

byshiue commented 1 year ago

Can you try 22.12, which is recommended in document?

18810251126 commented 1 year ago

This issue doesn't seem to be a version issue