This enhancement is to add support for LLMs running locally in the cluster (i.e., models and inferencing endpoints deployed by KAITO).
To use local LLMs, you will need to set the USE_LOCAL_LLM environment variable in the ai-service.yaml manifest to "True" and AI_ENDPOINT to the inferencing service that is running in the cluster
Does this introduce a breaking change?
[ ] Yes
[x] No
Pull Request Type
What kind of change does this Pull Request introduce?
[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:
To test local LLM, deploy KAITO to AKS cluster and update ai-service.yaml manifest to set USE_LOCAL_LLM to "True" and AI_ENDPOINT to the inferencing service that KAITO configures. Also be sure to unset the OpenAI specific environment variables (e.g., USE_AZURE_OPENAI, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT_NAME, OPENAI_API_KEY, and OPENAI_ORG_ID in the ai-service.yaml manifest)
Purpose
USE_LOCAL_LLM
environment variable in the ai-service.yaml manifest to "True" andAI_ENDPOINT
to the inferencing service that is running in the clusterDoes this introduce a breaking change?
Pull Request Type
What kind of change does this Pull Request introduce?
How to Test
ai-service.yaml
manifest to setUSE_LOCAL_LLM
to"True"
andAI_ENDPOINT
to the inferencing service that KAITO configures. Also be sure to unset the OpenAI specific environment variables (e.g.,USE_AZURE_OPENAI
,AZURE_OPENAI_ENDPOINT
,AZURE_OPENAI_DEPLOYMENT_NAME
,OPENAI_API_KEY
, andOPENAI_ORG_ID
in theai-service.yaml
manifest)