Add instructions about setting TGI(S) parameters

opendatahub-io / caikit-tgis-serving

Apache License 2.0

17 stars 39 forks source link

Add instructions about setting TGI(S) parameters #74

Open bdattoma opened 10 months ago

bdattoma commented 10 months ago

Users may need to set particular TGI(S) parameter when using Caikit+TGIS runtime on KServe. An example is the model timeout parameter which can be necessary to be tweaked based on the model size.

We should document the procedure in our docs

Additionally, in a future UI effort, this option should be present in the user interface

bdattoma commented 10 months ago

@MelissaFlinn @Jooho

dtrifiro commented 9 months ago

Related #115