search

deepjavalibrary / djl-serving

A universal scalable machine learning model deployment solution
Apache License 2.0
195 stars 64 forks source link

awscurl: WARN maxLength is not explicitly specified, use modelMaxLength: 512 #2339

Open CoolFish88 opened 1 month ago

CoolFish88 commented 1 month ago

Description

When requesting tokens per second in benchmark metrics (-t option specified) while providing the path to the tokenizer.json file as well as a payloads dataset, aws curl return the warning: WARN maxLength is not explicitly specified, use modelMaxLength: 512

I used a vLLM engine for which OPTION_MAX_MODEL_LEN was set to a higher value than 512.

(1) Are modelMaxLength and OPTION_MAX_MODEL_LEN? (2) Is modelMaxLength overwriting OPTION_MAX_MODEL_LEN? (3) Are token metrics impacted by this truncation?

Remark: A "tps" metric (requests per second) is included in the benchmark response. Since "tps" is usually used as an acronym for Tokens per second within the inference domain, requests per second could benefit from another abbreviation. Does "tokenThroughput" represent in this case tokens per second?

Expected Behavior

Use model server parameter as opposed to setting it to a default value

Error Message

WARN maxLength is not explicitly specified, use modelMaxLength: 512

How to Reproduce?

TOKENIZER= ./awscurl -c 1 -N 10 -X POST -n sagemaker --dataset -H 'Content-Type: application/json' -P --connect-timeout 60 -t

frankfliu commented 1 month ago

The warning is just for the awscurl tool. awscurl uses the tokenizer to calculate the number of output token.

I created a PR that allows you to configure the max length with env var: https://github.com/deepjavalibrary/djl-serving/pull/2341

sindhuvahinis commented 3 weeks ago

@CoolFish88 The PR is now merged. Let us know if you need anything else.