search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.28k stars 243 forks source link

Is seed actually used? #220

Closed richardburleigh closed 7 months ago

richardburleigh commented 7 months ago

Thank you for this amazing project!

Looking at the ExLlamaV2BaseGenerator code, I can't find where the seed parameter is actually passed to the generator:

https://github.com/turboderp/exllamav2/blob/5c974259bd245ace74ba4e8dda319d1f87d04c70/exllamav2/generator/base.py#L65-L67

VldmrB commented 7 months ago

Yes, here, for example:

https://github.com/turboderp/exllamav2/blob/5c974259bd245ace74ba4e8dda319d1f87d04c70/exllamav2/generator/streaming.py#L312

random.seed seeds the random generator to produce random values based on the given seed (after one generation, the seed is discarded), and the same value will be produced, provided the same seed is set before each call to random.random() (or similar funcs).

However, this is only present in generate_simple, and is not exposed in the Streaming generator, it seems, which is the more used/useful one. Perhaps it'd be easier to have it set directly on the sampler as an attribute, and check it inside the sample func.