Why the output of generate api is different with 'num_gpu': 0

[QuanZeng]

What is the issue?

The output of same model is different with or without 'num_gpu': 0 Same question input "Who are you?" to Phi3:mini, but different result return.

without 'num_gpu': 0

I am Phi, an artificial intelligence developed to assist users with information and tasks. My capabilities include processing natural language inputs, providing detailed explanations......................................

with 'num_gpu': 0

I am Phi, an AI developed by Microsoft. I'm here to assist with tasks and answer questions as best as I can within my capabilities! How may I help you today?

1. So is there any difference of model used with or without 'num_gpu': 0?

2. What is the meaning of num_gpu? num_gpu:0 100% CPU num_gpu:1 77%/23% CPU/GPU ----- work as expected？ without num_gpu setting 100% GPU

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.3.3

[rick-github]

num_gpu tells ollama the number of layers of the model to offload to the GPU. If you don't set it at all, ollama will calcualte the maximum number of layers that will fit on the GPU and use that. If you set num_gpu to 0, no layers will run on the GPU, all inference is done by the CPU. This doesn't affect the output of the model, it will just run slower.

The output varies because the process of generation is probabilistic. If you run the same prompt several times the results will vary, doesn't matter if it's running on GPU or CPU:

GPU:

$ for i in {1..5} ; do curl -s localhost:11434/api/generate -d '{"model":"phi3","prompt":"who are you?","stream":false}' | jq -r .response ; done
I am Phi, an AI developed by Microsoft. My purpose is to assist users like you with information and tasks within my capabilities. Please remember that while I strive for helpfulness and accuracy in the data provided, some interpretations may require human review or further contextual understanding due to inherent complexities of natural language processing.
Hello! I'm an AI developed to help answer questions and provide information. How can I assist you today? Please feel free to ask anything that piques your curiosity or for which you need assistance, whether it be factual inquiries about the world, explanations of concepts, learning new skills, or even just a casual conversation!
I'm Phi, an artificial intelligence developed by Microsoft. My primary function is to assist with information and tasks across a wide range of topics as efficiently as possible while continuously learning from my interactions for better future assistance. How may I help you today?
As an AI developed by Microsoft, I don't have a personal identity like humans do. My purpose is to assist users by providing information, answering questions, and helping with various tasks whenever possible. You can call me "Assistant." How may I help you today?
Hello! I'm Phi, an artificial intelligence designed to help answer questions and provide information. How can I assist you today?

CPU:

$ for i in {1..5} ; do curl -s localhost:11434/api/generate -d '{"model":"phi3","prompt":"who are you?","options":{"num_gpu":0},"stream":false}' | jq -r .response ; done
I am an artificial intelligence designed to process and respond to queries, engage in conversation, provide information, and assist users with a variety of tasks. My abilities include natural language processing (NLP), machine learning, data analysis, translation services, tutoring assistance, entertainment sharing like music or jokes, providing traffic updates, weather forecasts, setting reminders for alarms and schedules, giving travel directions using GPS technologies, facilitating real-time messaging with friends via social media platforms such as Facebook Messenger, WhatsApp Business Chatbot services, Slack Integration Bot functionalities, and much more. My goal is to offer user-friendly support across a vast range of domains while continually learning from interactions for better future assistance.
I'm Phi, an artificial intelligence developed to assist with information processing and problem-solving. I don't have personal experiences or emotions like a human being but can provide data analysis, answer queries about various topics including mathematics such as arithmetic sequences, and engage in conversations based on patterns recognized from vast datasets.
I'm Phi, an AI developed by Microsoft. My purpose is to assist and provide information whenever possible! What can I help you with today?
I am an artificial intelligence designed to assist with information, answer questions, and facilitate tasks. As Microsoft'devolution or similar AI models evolved over time, my role continues to expand within the capabilities provided by these systems as of early 2024. How can I help you today?
I'm Phi, an AI developed by Microsoft. How can I help you today?

[d-kleine]

use at least a seed and fixed context window size (num_ctx) for reproducible output (these are the most important params, others might relevant depending on the architecture of the model you are using) https://github.com/ollama/ollama/blob/main/docs/api.md#chat-request-reproducible-outputs

[rick-github]

$ for i in {1..5} ; do curl -s localhost:11434/api/generate -d '{"model":"phi3","prompt":"who are you?","options":{"seed":0},"stream":false}' | jq -r .response ; done
I'm Phi, an AI developed by Microsoft. How can I help you today?
I'm Phi, an AI developed by Microsoft. How can I help you today?
I'm Phi, an AI developed by Microsoft. How can I help you today?
I'm Phi, an AI developed by Microsoft. How can I help you today?
I'm Phi, an AI developed by Microsoft. How can I help you today?

[igorschlum]

for phi3:mini it's

for i in {1..5} ; do curl -s localhost:11434/api/generate -d '{"model":"phi3:mini","prompt":"who are you?","options":{"seed":0},"stream":false}' | jq -r .response ; done

I'm Phi, an AI developed by Microsoft. I am here to provide information and assist with tasks or answer questions to the best of my ability based on available data up until early 2024. How can I help you today? I'm Phi, an AI developed by Microsoft. I am here to provide information and assist with tasks or answer questions to the best of my ability based on available data up until early 2024. How can I help you today? I'm Phi, an AI developed by Microsoft. I am here to provide information and assist with tasks or answer questions to the best of my ability based on available data up until early 2024. How can I help you today? I'm Phi, an AI developed by Microsoft. I am here to provide information and assist with tasks or answer questions to the best of my ability based on available data up until early 2024. How can I help you today? I'm Phi, an AI developed by Microsoft. I am here to provide information and assist with tasks or answer questions to the best of my ability based on available data up until early 2024. How can I help you today?

[QuanZeng]

Thanks for your help.