Open parfeniukink opened 2 months ago
The Deepsparse Backend interface is implemented.
pyproject.toml
ruff
DeepsparseBackend
settings.py::DeepsparseSettings
tests/unit/backend/deepsparse.py
TestTextGenerationPipeline
deepsparse.pipeline.Pipeline
This is an example of a command you can use in your terminal:
--data=openai_humaneval
--model=/local/path/my_model
python -m src.guidellm.main --data=openai_humaneval --max-requests=1 --max-seconds=20 --rate-type=constant --rate=1.0 --backend=deepsparse --model=/local-path
The model could also be set with GUIDELLM__LLM_MODEL. If the CLI value or environment variable is not set, then the default will be used. Currently, the default model is: mistralai/Mistral-7B-Instruct-v0.3.
GUIDELLM__LLM_MODEL
mistralai/Mistral-7B-Instruct-v0.3
Summary
The Deepsparse Backend interface is implemented.
pyproject.tomlfilerufferrors are suppressedDeepsparseBackendgets configurations from CLI, environment or defaultssettings.py::DeepsparseSettingsincludes all the settingstests/unit/backend/deepsparse.pyincludes unit testsTestTextGenerationPipelinemocks thedeepsparse.pipeline.PipelineUsage
This is an example of a command you can use in your terminal:
--data=openai_humaneval: determines the dataset--model=/local/path/my_model: determines the local path to the model object. If not specified - the env variable will be used.Environment configuration
The model could also be set with
GUIDELLM__LLM_MODEL. If the CLI value or environment variable is not set, then the default will be used. Currently, the default model is:mistralai/Mistral-7B-Instruct-v0.3.