fix infinite generation

openvinotoolkit / openvino.genai

Run Generative AI models using native OpenVINO C++ API

Apache License 2.0

113 stars 151 forks source link

fix infinite generation #563

Closed pavel-esir closed 3 months ago

pavel-esir commented 3 months ago

It turned out that if user didn't specify config in generation then a default GenerationConfig is constructed which wipes out config red from genration_config.json.

Pass std::nullopt if user didn't specify genration config.

Updated docs in openvino repo as well https://github.com/openvinotoolkit/openvino/pull/25283

Tickets: CVS-145154 CVS-145218