search

huggingface / optimum-nvidia

Apache License 2.0
889 stars 85 forks source link

MSL default in model config #6

Closed laikhtewari closed 11 months ago

laikhtewari commented 11 months ago

For model repos that don't have max_sequence_length in the config.json, default to usingmax_position_embeddings. If max_position_embeddings is also not present, throw an error

laikhtewari commented 11 months ago

You still need to specify the max_new_tokens from the pipeline definition. Can we default to MSL if not user-specified?

mfuntowicz commented 11 months ago

You still need to specify the max_new_tokens from the pipeline definition. Can we default to MSL if not user-specified?

It should be automatically computed from the max_sequence_length - max_prompt_length, see here