search

triton-inference-server / fastertransformer_backend

BSD 3-Clause "New" or "Revised" License
411 stars 134 forks source link

will FT5.0 be supported ? #15

Closed 520jefferson closed 2 years ago

520jefferson commented 2 years ago

@yuanzhedong @PerkzZheng @GwangsooHong @deadeyegoodwin @byshiue Is there any plan to support FT5.0, some model structures like T5 only exist on FT5.0. https://github.com/NVIDIA/FasterTransformer/blob/dev/v5.0_beta/src/fastertransformer/triton_backend/t5/CMakeLists.txt

byshiue commented 2 years ago

FT triton backend already supports T5, please refer https://github.com/triton-inference-server/fastertransformer_backend/tree/dev/v1.1_beta#prepare-triton-t5-model-store-in-the-docker.

520jefferson commented 2 years ago

I do as README， but i met some errors as follows: Firstly, i start the server(the model path cannot be setted like --model-repository=/workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/ or --model-repository=/workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1 , because the program will failed with -- error: creating server: Internal - failed to load all models) as follows: mpirun -n 1 --allow-run-as-root /opt/tritonserver/bin/tritonserver --model-repository=/workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1/2-gpu/ image image

Secondly, i start the client sh /workspace/fastertransformer_backend/tools/run_client.sh then i get this in logs: image

it looks like that i haven't load the config.pbtxt, how should i load it? i put the default t5 config.pbtxt in both /workspace/build/fastertransformer_backend/all_models/t5/fastertransformer and /workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1 and /workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1/2-gpu, it doesn't work, the client cann't identity the model name fastertransformer.