load failed for model - Githubissues

System Info

GPU Nvidia A10G Cuda version 12.3 Driver version 535.183.01 TensorRT-LLM v0.8.0 Image nvcr.io/nvidia/tritonserver:24.02-trtllm-python-py3 (was used to build the tensorrt engine and start triton inference server) Model meta-llm/Meta-Llama-Guard-2-8B OS: Ubuntu

Who can help?

@byshiue @nv-guomingz @hijkzzz

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

convert model checkpoint to tensorrt format: python3 convert_checkpoint.py ...
build tensorrt engine: trtllm-build ...
run inference with tensorrt engine: python3 run.py ...
create trt model repo based on .../tensorrtllmbackend/all_models/inflight_batcher_llm/
run triton inference server (see debug.txt)

debug.txt: debug_trt_llm.txt

cat /tensorrt/triton-repos/trt-Meta-Llama-Guard-2-8B/postprocessing/1/model.py

# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of NVIDIA CORPORATION nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json

import numpy as np
import triton_python_backend_utils as pb_utils
from transformers import AutoTokenizer, LlamaTokenizer, T5Tokenizer

class TritonPythonModel:
    """Your Python model must use the same class name. Every Python model
    that is created must have "TritonPythonModel" as the class name.
    """

    def initialize(self, args):
        """`initialize` is called only once when the model is being loaded.
        Implementing `initialize` function is optional. This function allows
        the model to initialize any state associated with this model.
        Parameters
        ----------
        args : dict
          Both keys and values are strings. The dictionary keys and values are:
          * model_config: A JSON string containing the model configuration
          * model_instance_kind: A string containing model instance kind
          * model_instance_device_id: A string containing model instance device ID
          * model_repository: Model repository path
          * model_version: Model version
          * model_name: Model name
        """
        # Parse model configs
        model_config = json.loads(args['model_config'])
        tokenizer_dir = model_config['parameters']['tokenizer_dir'][
            'string_value']
        tokenizer_type = model_config['parameters']['tokenizer_type'][
            'string_value']
        self.skip_special_tokens = model_config['parameters'].get(
            'skip_special_tokens',
            {'string_value': "true"})['string_value'].lower() in [
                'true', '1', 't', 'y', 'yes'
            ]

        if tokenizer_type == 't5':
            self.tokenizer = T5Tokenizer(vocab_file=tokenizer_dir,
                                         padding_side='left')
        elif tokenizer_type == 'auto':
            self.tokenizer = AutoTokenizer.from_pretrained(
                tokenizer_dir, padding_side='left', trust_remote_code=True)
        elif tokenizer_type == 'llama':
            self.tokenizer = LlamaTokenizer.from_pretrained(
                tokenizer_dir, legacy=False, padding_side='left')
        else:
            raise AttributeError(
                f'Unexpected tokenizer type: {tokenizer_type}')
        self.tokenizer.pad_token = self.tokenizer.eos_token

        # Parse model output configs
        output_config = pb_utils.get_output_config_by_name(
            model_config, "OUTPUT")

        # Convert Triton types to numpy types
        self.output_dtype = pb_utils.triton_string_to_numpy(
            output_config['data_type'])

    def execute(self, requests):
        """`execute` must be implemented in every Python model. `execute`
        function receives a list of pb_utils.InferenceRequest as the only
        argument. This function is called when an inference is requested
        for this model. Depending on the batching configuration (e.g. Dynamic
        Batching) used, `requests` may contain multiple requests. Every
        Python model, must create one pb_utils.InferenceResponse for every
        pb_utils.InferenceRequest in `requests`. If there is an error, you can
        set the error argument when creating a pb_utils.InferenceResponse.
        Parameters
        ----------
        requests : list
          A list of pb_utils.InferenceRequest
        Returns
        -------
        list
          A list of pb_utils.InferenceResponse. The length of this list must
          be the same as `requests`
        """

        responses = []

        # Every Python backend must iterate over everyone of the requests
        # and create a pb_utils.InferenceResponse for each of them.
        for idx, request in enumerate(requests):
            # Get input tensors
            tokens_batch = pb_utils.get_input_tensor_by_name(
                request, 'TOKENS_BATCH').as_numpy()

            # Get sequence length
            sequence_lengths = pb_utils.get_input_tensor_by_name(
                request, 'SEQUENCE_LENGTH').as_numpy()

            # Get cum log probs
            cum_log_probs = pb_utils.get_input_tensor_by_name(
                request, 'CUM_LOG_PROBS').as_numpy()

            # Get sequence length
            output_log_probs = pb_utils.get_input_tensor_by_name(
                request, 'OUTPUT_LOG_PROBS').as_numpy()

            # Get context logits
            context_logits = pb_utils.get_input_tensor_by_name(
                request, 'CONTEXT_LOGITS').as_numpy()

            # Get generation logits
            generation_logits = pb_utils.get_input_tensor_by_name(
                request, 'GENERATION_LOGITS').as_numpy()

            # Reshape Input
            # tokens_batch = tokens_batch.reshape([-1, tokens_batch.shape[0]])
            # tokens_batch = tokens_batch.T

            # Postprocessing output data.
            outputs = self._postprocessing(tokens_batch, sequence_lengths)

            # Create output tensors. You need pb_utils.Tensor
            # objects to create pb_utils.InferenceResponse.
            output_tensor = pb_utils.Tensor(
                'OUTPUT',
                np.array(outputs).astype(self.output_dtype))

            out_cum_log_probs = pb_utils.Tensor('OUT_CUM_LOG_PROBS',
                                                cum_log_probs)

            out_output_log_probs = pb_utils.Tensor('OUT_OUTPUT_LOG_PROBS',
                                                   output_log_probs)

            out_context_logits = pb_utils.Tensor('OUT_CONTEXT_LOGITS',
                                                 context_logits)

            out_generation_logits = pb_utils.Tensor('OUT_GENERATION_LOGITS',
                                                    generation_logits)

            # Create InferenceResponse. You can set an error here in case
            # there was a problem with handling this inference request.
            # Below is an example of how you can set errors in inference
            # response:
            #
            # pb_utils.InferenceResponse(
            #    output_tensors=..., TritonError("An error occurred"))
            inference_response = pb_utils.InferenceResponse(output_tensors=[
                output_tensor, out_cum_log_probs, out_output_log_probs,
                out_context_logits, out_generation_logits
            ])
            responses.append(inference_response)

        # You should return a list of pb_utils.InferenceResponse. Length
        # of this list must match the length of `requests` list.
        return responses

    def finalize(self):
        """`finalize` is called only once when the model is being unloaded.
        Implementing `finalize` function is optional. This function allows
        the model to perform any necessary clean ups before exit.
        """
        print('Cleaning up...')

    def _postprocessing(self, tokens_batch, sequence_lengths):
        outputs = []
        for batch_idx, beam_tokens in enumerate(tokens_batch):
            for beam_idx, tokens in enumerate(beam_tokens):
                seq_len = sequence_lengths[batch_idx][beam_idx]
                output = self.tokenizer.decode(
                    tokens[:seq_len],
                    skip_special_tokens=self.skip_special_tokens)
                outputs.append(output.encode('utf8'))
        return outputs

cat /tensorrt/triton-repos/trt-Meta-Llama-Guard-2-8B/postprocessing/config.pbtxt

# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of NVIDIA CORPORATION nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

name: "postprocessing"
backend: "python"
max_batch_size: 64
input [
  {
    name: "TOKENS_BATCH"
    data_type: TYPE_INT32
    dims: [ -1, -1 ]
  },
  {
    name: "SEQUENCE_LENGTH"
    data_type: TYPE_INT32
    dims: [ -1 ]
  },
  {
    name: "CUM_LOG_PROBS"
    data_type: TYPE_FP32
    dims: [ -1 ]
  },
  {
    name: "OUTPUT_LOG_PROBS"
    data_type: TYPE_FP32
    dims: [ -1, -1 ]
  },
  {
    name: "CONTEXT_LOGITS"
    data_type: TYPE_FP32
    dims: [ -1, -1 ]
    optional: true
  },
  {
    name: "GENERATION_LOGITS"
    data_type: TYPE_FP32
    dims: [ -1, -1, -1 ]
    optional: true
  }
]
output [
  {
    name: "OUTPUT"
    data_type: TYPE_STRING
    dims: [ -1 ]
  },
  {
    name: "OUT_CUM_LOG_PROBS"
    data_type: TYPE_FP32
    dims: [ -1 ]
  },
  {
    name: "OUT_OUTPUT_LOG_PROBS"
    data_type: TYPE_FP32
    dims: [ -1, -1 ]
  },
  {
    name: "OUT_CONTEXT_LOGITS"
    data_type: TYPE_FP32
    dims: [ -1, -1 ]
  },
  {
    name: "OUT_GENERATION_LOGITS"
    data_type: TYPE_FP32
    dims: [ -1, -1, -1 ]
  }
]

parameters {
  key: "tokenizer_dir"
  value: {
    string_value: "/tensorrt/models/Meta-Llama-Guard-2-8B"
  }
}

parameters {
  key: "tokenizer_type"
  value: {
    string_value: "llama"
  }
}

parameters {
  key: "skip_special_tokens"
  value: {
    string_value: "True"
  }
}

instance_group [
    {
        count: 1
        kind: KIND_CPU
    }
]

NVIDIA / TensorRT-LLM

load failed for model #1859

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

actual behavior

additional notes