Currently servings that use random numbers accept a :seed option when building the serving, but users of LLMs actually expect each call to give a different reply. We can accept seed as part of serving input and pass that to generation.
The default behaviour should likely be that we actually always use a different seed. With that, we no longer need the :seed option on the serving.
Currently servings that use random numbers accept a
:seedoption when building the serving, but users of LLMs actually expect each call to give a different reply. We can accept seed as part of serving input and pass that to generation.The default behaviour should likely be that we actually always use a different seed. With that, we no longer need the
:seedoption on the serving.