search

huggingface / text-generation-inference

Large Language Model Text Generation Inference
http://hf.co/docs/text-generation-inference
Apache License 2.0
8.48k stars 968 forks source link

'temperature' must be strictly positive #1236

Closed austinmw closed 6 months ago

austinmw commented 8 months ago

System Info

SageMaker DLC: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.1.0-gpu-py39-cu118-ubuntu20.04

Information

Tasks

Reproduction

Deploying Mistral 7B and attempting to pass temperature parameter of 0.0.

Expected behavior

0 should be a valid temperature, and is desirable for many use cases, however, an error is produced if it is not strictly positive.

CloudWatch logs:

033[2mtext_generation_router::infer#033[0m#033[2m:#033[0m #033[2mrouter/src/infer.rs#033[0m#033[2m:#033[0m#033[2m111:#033[0m temperature must be strictly positive

OlivierDehaene commented 8 months ago

In what case a temperature of 0 is desirable?

github-actions[bot] commented 7 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

austinmw commented 7 months ago

@OlivierDehaene When you want a deterministic output

github-actions[bot] commented 6 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Legion2 commented 2 months ago

We are facing the same problem, we want a deterministic output for our fine tuned model, no randomeness.