I followed the recipe given here to manually convert teknium/OpenHermes-2.5-Mistral-7B to tensorrt on sagemaker's ml.g5.4xlarge and deploy the compiled model saved on s3 on sagemaker endpoint using ml.g5.2xlarge (only cpu and ram are different). When i invoke the endpoint simply using
I assume the error is coming from giving a list of lists to the _tokenizer.decode function instead of just a list of input_ids. Can someone help me understand why this happens ?
Description
I followed the recipe given here to manually convert teknium/OpenHermes-2.5-Mistral-7B to tensorrt on sagemaker's
ml.g5.4xlarge
and deploy the compiled model saved on s3 on sagemaker endpoint usingml.g5.2xlarge
(only cpu and ram are different). When i invoke the endpoint simply usingI receive the following error log:
Error Message
I assume the error is coming from giving a list of lists to the _tokenizer.decode function instead of just a list of input_ids. Can someone help me understand why this happens ?