this expands the public api with a Model class and register_model function.
background: we currently support hosted (base_url of integration.api.nvidia.com/v1) as well as local NIMs (base_url controlled by user, e.g. localhost:1234/v1) using the base_url feature. the base_url must provide paths for model listing, /models, and inference, one of /chat/completion, /embeddings or /ranking.
use case: users would like to support endpoints that only support inference (do not support model listing) and to minimize code changes when model names stay the same and endpoints change.
from langchain_nvidia_ai_endpoints import ChatNVIDIA
llm = ChatNVIDIA(model="my-custom-model-name", endpoint="http://host:port/path-to-my-model")
reason for rejection: endpoints will change over time resulting in multiple refactoring points in user code. accepted design allows users to centrally add a model and then refer to it throughout their code is simpler.
this expands the public api with a
Model
class andregister_model
function.background: we currently support hosted (base_url of integration.api.nvidia.com/v1) as well as local NIMs (base_url controlled by user, e.g. localhost:1234/v1) using the
base_url
feature. the base_url must provide paths for model listing,/models
, and inference, one of/chat/completion
,/embeddings
or/ranking
.use case: users would like to support endpoints that only support inference (do not support model listing) and to minimize code changes when model names stay the same and endpoints change.
design -
rejected design -
reason for rejection: endpoints will change over time resulting in multiple refactoring points in user code. accepted design allows users to centrally add a model and then refer to it throughout their code is simpler.