Mistral 7b instruct not working in pet store for local LLM

[lukemurraynz]

Describe the bug

Hi, all. Working on a blog article, following a mix of local documentation + Intelligent app workshop, but instead of going Falcon, I've gone with the Mistral 7b model. and at the end - the switch of the pet store app to use it.

I can prompt the model locally from the cluster using:

kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST "http://workspace-mistral-7b-instruct/generate" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"prompt\":\"What is your fav orite ice cream flavor?\"}"

However, using a pet store is not working, and it responds with 'Production Description' text instead.

Screenshots

This is my config map:

kubectl apply -n pets -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: ai-service-configmap data: USE_LOCAL_LLM: "True" AI_ENDPOINT: "http://workspace-mistral-7b-instruct/chat"

apiVersion: apps/v1 kind: Deployment metadata: name: ai-service spec: replicas: 1 selector: matchLabels: app: ai-service template: metadata: labels: app: ai-service spec: nodeSelector: "kubernetes.io/os": linux containers:

• name: order-service image: ghcr.io/azure-samples/aks-store-demo/ai-service:latest ports:
  • containerPort: 5001 envFrom:

  • configMapRef: name: ai-service-configmap resources: requests: cpu: 20m memory: 50Mi limits: cpu: 30m memory: 85Mi startupProbe: httpGet: path: /health port: 5001 initialDelaySeconds: 60 failureThreshold: 3 timeoutSeconds: 3 periodSeconds: 5 readinessProbe: httpGet: path: /health port: 5001 initialDelaySeconds: 3 failureThreshold: 3 timeoutSeconds: 3 periodSeconds: 5 livenessProbe: httpGet: path: /health port: 5001 failureThreshold: 3 initialDelaySeconds: 3 timeoutSeconds: 3 periodSeconds: 3

    apiVersion: v1 kind: Service metadata: name: ai-service spec: type: ClusterIP ports:

    • name: http port: 5001 targetPort: 5001 selector: app: ai-service EOF

Hoping someone can point me in the direction on whats happening here, whether its a bug, or what needs changing.

[pauldotyu]

Thanks for raising this issue @lukemurraynz! Will take a closer look and let you know.