This PR adds support for respecting a TGIS hostname override (x-route-info header/metadata).
This is achieved by lazily registering the model connection with the TGISBackend when .run()/run_stream_out()/run_tokenizer() is called. The x-route-info header/metadata is read from the context arg.
Additionally:
a TextGenerationTGIS._tgis_backend property was added to be consistent with PeftPromptTuningTGIS. The TextGenerationTGIS.tgis_backend property is conditional on the tgis_backend argument to be True, which made accessing TextGenerationTGIS.tgis_backend risky as it might not exist. The additional _tgis_backend property is guaranteed to exist, making using it simpler and less risky.
This PR adds support for respecting a TGIS hostname override (
x-route-infoheader/metadata).This is achieved by lazily registering the model connection with the TGISBackend when
.run()/run_stream_out()/run_tokenizer()is called. Thex-route-infoheader/metadata is read from the context arg.Additionally:
TextGenerationTGIS._tgis_backendproperty was added to be consistent withPeftPromptTuningTGIS. TheTextGenerationTGIS.tgis_backendproperty is conditional on thetgis_backendargument to be True, which made accessingTextGenerationTGIS.tgis_backendrisky as it might not exist. The additional_tgis_backendproperty is guaranteed to exist, making using it simpler and less risky.Depends on
caikit-tgis-backend>=0.1.33