Open parfeniukink opened 2 months ago
This PR extends the PR: Deepsparse Backend implementation. The base branch is parfeniukink/features/deepsparse-backend.
parfeniukink/features/deepsparse-backend
vllm
VllmBackend
guidellm/backend/vllm
This is an example of a command you can use in your terminal:
--data=openai_humaneval
--model=/local/path/my_model
python -m src.guidellm.main --data=openai_humaneval --max-requests=1 --max-seconds=20 --rate-type=constant --rate=1.0 --backend=vllm --model=/local-path
The model could also be set with GUIDELLM__LLM_MODEL. If the CLI value or environment variable is not set, then the default will be used. Currently, the default model is: mistralai/Mistral-7B-Instruct-v0.3.
GUIDELLM__LLM_MODEL
mistralai/Mistral-7B-Instruct-v0.3
Summary
This PR extends the PR: Deepsparse Backend implementation. The base branch is
parfeniukink/features/deepsparse-backend.vllmis added to optional dependenciesVllmBackendclass encapsulates the vLLM integration.guidellm/backend/vllmis available only if the Python version and the runtime platform pass the validation.vllmtests are skipped in case the platform is not LinuxUsage
This is an example of a command you can use in your terminal:
--data=openai_humaneval: determines the dataset--model=/local/path/my_model: determines the local path to the model object. If not specified - the env variable will be used.Environment configuration
The model could also be set with
GUIDELLM__LLM_MODEL. If the CLI value or environment variable is not set, then the default will be used. Currently, the default model is:mistralai/Mistral-7B-Instruct-v0.3.