Hi, when I ensemble a fastertransformer_backend GPT model, it loaded the ensemble model failed with the error when starting the server. Could you please give some advice? Thanks.
CUDA_VISIBLE_DEVICES="0,1" /opt/tritonserver/bin/tritonserver --model-store=fastertransformer_backend/all_models/nemo-megatron-gpt-5B > log 2>&1 &
...
E0613 07:51:38.390571 142822 model_repository_manager.cc:1002] Poll failed for model directory 'ensemble': output 'OUTPUT_0' for ensemble 'ensemble' is not written
...
I0613 07:51:48.643366 142822 server.cc:264] Waiting for in-flight requests to complete.
I0613 07:51:48.643407 142822 server.cc:280] Timeout 30: Found 0 model versions that have in-flight inferences
I0613 07:51:48.643479 142822 server.cc:295] All models are stopped, unloading models
I0613 07:51:48.643495 142822 server.cc:302] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I0613 07:51:48.643572 142822 libfastertransformer.cc:1965] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0613 07:51:48.643651 142822 libfastertransformer.cc:1899] TRITONBACKEND_ModelFinalize: delete model state
I0613 07:51:48.643688 142822 libfastertransformer.cc:1904] TRITONBACKEND_ModelFinalize: MPI Finalize
I0613 07:51:48.730007 142822 model_lifecycle.cc:579] successfully unloaded 'fastertransformer' version 1
I0613 07:51:49.643671 142822 server.cc:302] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
In my case, I only ensemble the fastertransformer model. My model resposity fastertransformer_backend/all_models/nemo-megatron-gpt-5B includes ensemble and fastertransformer folders.
Hi, when I ensemble a fastertransformer_backend GPT model, it loaded the ensemble model failed with the error when starting the server. Could you please give some advice? Thanks.
Reproduce: model weight: https://huggingface.co/nvidia/nemo-megatron-gpt-5B ensemble example: https://github.com/triton-inference-server/fastertransformer_backend/tree/main/all_models/gpt
In my case, I only ensemble the fastertransformer model. My model resposity
fastertransformer_backend/all_models/nemo-megatron-gpt-5B
includesensemble
andfastertransformer
folders.