ERROR: expected number of inputs between 1 and 3 but got 9 inputs for model

[samzong]

{"timestamp":"2024-04-15T05:20:55.796456Z","level":"ERROR","error":"AppError(error message received from triton: [request id: ] expected number of inputs between 1 and 3 but got 9 inputs for model 'myserving')","target":"openai_trtllm::routes::completions","span":{"headers":"{\"host\": \"localhost:3030\", \"user-agent\": \"OpenAI/Python 1.17.1\", \"content-length\": \"55\", \"accept\": \"application/json\", \"accept-encoding\": \"gzip, deflate\", \"authorization\": \"Bearer test\", \"content-type\": \"application/json\", \"x-stainless-arch\": \"arm64\", \"x-stainless-async\": \"false\", \"x-stainless-lang\": \"python\", \"x-stainless-os\": \"MacOS\", \"x-stainless-package-version\": \"1.17.1\", \"x-stainless-runtime\": \"CPython\", \"x-stainless-runtime-version\": \"3.10.5\"}","name":"non-streaming completions"},"spans":[{"http.request.method":"POST","http.route":"/v1/completions","network.protocol.version":"1.1","otel.kind":"Server","otel.name":"POST /v1/completions","server.address":"localhost:3030","span.type":"web","url.path":"/v1/completions","url.scheme":"","user_agent.original":"OpenAI/Python 1.17.1","name":"HTTP request"},{"headers":"{\"host\": \"localhost:3030\", \"user-agent\": \"OpenAI/Python 1.17.1\", \"content-length\": \"55\", \"accept\": \"application/json\", \"accept-encoding\": \"gzip, deflate\", \"authorization\": \"Bearer test\", \"content-type\": \"application/json\", \"x-stainless-arch\": \"arm64\", \"x-stainless-async\": \"false\", \"x-stainless-lang\": \"python\", \"x-stainless-os\": \"MacOS\", \"x-stainless-package-version\": \"1.17.1\", \"x-stainless-runtime\": \"CPython\", \"x-stainless-runtime-version\": \"3.10.5\"}","name":"completions"},{"headers":"{\"host\": \"localhost:3030\", \"user-agent\": \"OpenAI/Python 1.17.1\", \"content-length\": \"55\", \"accept\": \"application/json\", \"accept-encoding\": \"gzip, deflate\", \"authorization\": \"Bearer test\", \"content-type\": \"application/json\", \"x-stainless-arch\": \"arm64\", \"x-stainless-async\": \"false\", \"x-stainless-lang\": \"python\", \"x-stainless-os\": \"MacOS\", \"x-stainless-package-version\": \"1.17.1\", \"x-stainless-runtime\": \"CPython\", \"x-stainless-runtime-version\": \"3.10.5\"}","name":"non-streaming completions"}]}

use client/openai_completion.py

[samzong]

I think I know the problem, my trition backend use trition with vllm.

Do we have a plan to support it?

[npuichigo]

it's not planned yet, but I think it's trivial to adapt the codes for your use case.

[samzong]

it's not planned yet, but I think it's trivial to adapt the codes for your use case.

Do you have any suggestions? I can try to implement it, and if I can, I can contribute this part of the code

[npuichigo]

Can you provide how to calling vllm-based triton backend? The grpc interface, the parameters for example to call the service.

[samzong]

okay, @npuichigo You can see an example here.

https://github.com/triton-inference-server/vllm_backend/blob/a01475157290bdf6fd0f50688f69aafea41b04c5/samples/client.py#L192

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-m",
        "--model",
        type=str,
        required=False,
        default="vllm_model",
        help="Model name",
    )
    parser.add_argument(
        "-v",
        "--verbose",
        action="store_true",
        required=False,
        default=False,
        help="Enable verbose output",
    )
    parser.add_argument(
        "-u",
        "--url",
        type=str,
        required=False,
        default="localhost:8001",
        help="Inference server URL and its gRPC port. Default is localhost:8001.",
    )
    parser.add_argument(
        "-t",
        "--stream-timeout",
        type=float,
        required=False,
        default=None,
        help="Stream timeout in seconds. Default is None.",
    )
    parser.add_argument(
        "--offset",
        type=int,
        required=False,
        default=0,
        help="Add offset to request IDs used",
    )
    parser.add_argument(
        "--input-prompts",
        type=str,
        required=False,
        default="prompts.txt",
        help="Text file with input prompts",
    )
    parser.add_argument(
        "--results-file",
        type=str,
        required=False,
        default="results.txt",
        help="The file with output results",
    )
    parser.add_argument(
        "--iterations",
        type=int,
        required=False,
        default=1,
        help="Number of iterations through the prompts file",
    )
    parser.add_argument(
        "-s",
        "--streaming-mode",
        action="store_true",
        required=False,
        default=False,
        help="Enable streaming mode",
    )
    parser.add_argument(
        "--exclude-inputs-in-outputs",
        action="store_true",
        required=False,
        default=False,
        help="Exclude prompt from outputs",
    )

[liyan77]

my trition backend also use trition with vllm，have a plan to support it？

[crslen]

Would be great if vllm option was supported.

[ChaseDreamInfinity]

I made some change to let it support vllm backend. https://github.com/ChaseDreamInfinity/openai_triton_vllm