search

caikit / caikit-nlp

Apache License 2.0
12 stars 49 forks source link

generate_text_func: support `finish_reason=TOKEN_LIMIT` #253

Open dtrifiro opened 1 year ago

dtrifiro commented 1 year ago

generate_text_func currently does not correctly return finish_reason=TOKEN_LIMIT when reaching the model token limit:

TOKEN_LIMIT refers to the maximum number of tokens limit defined by the model whereas the MAX_TOKENS refers to the maximum number defined by the user. So one can reach TOKEN_LIMIT before MAX_TOKENS

_Originally posted by @gkumbhat in https://github.com/caikit/caikit-nlp/pull/210#discussion_r1374803238_