ollama / ollama

Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.
https://ollama.com
MIT License
95.53k stars 7.57k forks source link

Mistral instruction following doesn't work as should be when the prompt is lengthy #2478

Closed rsandx closed 8 months ago

rsandx commented 8 months ago

I'm working on a project using RAG, which requires a model to answer questions based on the provided context. Having tested mistral-7b-instruct-v0.2.Q5_K_M.gguf served by llamafile and Ollama, I found that the instruction following works well for both servers when the prompt is not lengthy; but when the prompt is lengthy, the llamafile server returns proper contents (although not as good as ChatGPT's but acceptable), the Ollama server returns contents that feel like ignoring the context or hallucinating. See examples below for comparison and reproducing the results.

  1. llamafile with short prompt: curl http://localhost:8888/completion -d '{"prompt": "You'\''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'\''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nwhat'\''s Speech Synthesizer?"}' the response: {"content":"\nThe Speech Synthesizer is a powerful tool from aiTransformer that can convert text into human-like speech in various languages and accents. It also has the capability to generate videos with a presenter speaking the text and lip-syncing the avatar. This tool can be used to create personalized talking greeting cards, as demonstrated in this video, and is integrated with VideoPlus Studio for adding subtitles, generating speeches, and creating lip-synced avatars for videos/documents/images.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf","n_ctx":32768,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf"},"tokens_cached":424,"tokens_evaluated":313,"tokens_predicted":112,"truncated":false}

  2. llamafile with long prompt: curl http://localhost:8888/completion -d '{"prompt": "You'\''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Cartoonizer transforms real-world pictures into cartoon-style images. It offers several cartoon styles to choose from, with sample effects provided, and more styles to be added over time. The tool is designed to make it easy to create fun and unique cartoon images directly from photos. For more information on the different cartoon styles, read our blog posts at here and here, and watch the demo video for usage tips.\nWith the Enhancer, you can easily enhance your photos and make them look clearer, sharper, and more professional. The tool offers several enhance types to choose from, including options to restore faces, and more types will be added over time. For more information on the different enhance types, read our blog post at here.\nThe Sketcher makes it easy to turn photos into sketches. Whether you'\''re an artist or just looking for a fun way to express your creativity, the Sketcher offers four different sketch styles, as well as two byproducts for finding the edges and smoothing the image (good for selfies). Additionally, a colored head sketch can be produced if the input image contains a face. With its simple interface, anyone can create sketches in no time, no need for any art skills or experience. So why not give it a try today and see what kind of sketches you can create!\nUsing single image super resolution algorithms, the Enlarger can upsize images at a scale of up to 8 (64 times of the original size). This can produce high-resolution images that are suitable for large prints. However, it'\''s important to note that high-quality 8x zoom tends to work on smaller images, and that larger upscaling can take longer to process.\nThe Filter adds special effects to your photos and videos, giving them a unique and special touch. You can choose from a range of filters to give your photos a different look, from subtle tweaks to bold and dramatic effects. These filters are simple to use and can be applied with just a few clicks.\nThe Stylizer transforms photos into works of art inspired by famous artists and styles. Simply provide the source image and the desired style image, the Stylizer will generate six stylized versions with varying style intensities. Whether you'\''re a professional artist or just looking to have some fun, the Stylizer is the perfect tool to turn your photos into unique and eye-catching works of art. You can select from predefined style images or upload your own custom style image. This gives you a wide range of possibilities for creating unique and interesting stylized images. Additionally, the Stylizer has an option to apply styles to the whole image or to specific regions, such as the foreground or background. While the foreground and background detection is not always accurate, experimenting can lead to some unexpected and interesting results.\nThe MultiStylizer offers a creative approach to image styling by combining multiple styles into a single image. With its semantic-aware neural network, the MultiStylizer automatically incorporates different styles based on the regional semantics of the source image, creating a unique and visually stunning result that is different from the traditional single style transfer. With the option to select a predefined or custom style for each style choice, and the ability to apply styles to the whole region or auto-detected foreground / background, the possibilities for creating stunning digital art are limitless.\nBased on the powerful deep learning, text-to-image model Stable Diffusion, the Super Stylizer can generate stunning detailed images conditioned on text descriptions, and can also use the generated images to stylize your picture with adjustable style strength. In order to produce meaningful results, it'\''s important to describe elements to be included or omitted from the output. For more information on this topic, read our blog posts at here and here. To make things easier, a list of frequently-used art styles and mediums are provided, and there is a demo video to show how the tool works. We also have a dedicated Prompt Builder to help you build text prompts intuitively or create random prompts in one click, making the process even simpler and more fun. The Prompt Builder lists 1000+ short prompts with sample images, including 500+ textual inversion terms verified to work here. Unlike some other platforms, the images you generated here are absolutely private.\nThe Prompt Builder allows you to easily create text prompts for the Stable Diffusion text-to-image model in the Super Stylizer by providing a list of special terms and often-used generic terms with sample images. This helps you build text prompts intuitively, and you can even generate random prompts with just one click or generate prompts from images. The supported terms include 500+ (and growing) special Textual Inversion terms, giving you a wider vocabulary to use in your text prompts.\nThe Background Editor can remove the background from an image, leaving only the subject of the image visible. The process uses machine learning algorithms to identify the subject and separate it from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, while also setting the transparency level for both the foreground and background. For more information on this topic, read our blog posts at here and here.\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'\''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nVideoPlus Studio first is a free subtitle editor and translator, that allows you to add subtitles to videos, documents and images, edit and translate them to other languages. Moreover, you can select a presenter for each subtitle, the presenter has properties for turning the text into speeches in certain language and voice, and an image (optional) for generating a lip-synced avatar to speak the text; the avatar has properties like shape, size and location to control how it'\''s going to show in the video. See some use cases and sample videos on this page. Besides, the app can also apply different cartoon styles and filters to videos, as well as transcribe audio to text with automatically detected language and save the text to a subtitle file that'\''s ready to use in this app.\nIf you just want to apply special effects to videos, open the task window by clicking the '\''Task'\'' button in the menu, then click the '\''Submit New Video Effect Task'\'' button, select your video and effect and submit to process. To get a video transcription, click the '\''Submit New Video Transcription Task'\'' button, select your video and submit to process. To add/edit subtitles with presenters, you should get familiar with 3 key concepts used in this app. Subtitles: Subtitles are text representing the contents of the audio in a video, they must include information on when each line of text should appear and disappear. The subtitle editor is on the main screen besides the video/document/image player. Subtitle editing for a video is based on timeline, while it is based on page number or image frame for a document or an image. By default the main screen is loaded with this tutorial video and its subtitles, you can play around with it to learn the subtitle editing features. Presenter: The Subtitle has a Presenter property, that'\''s used to turn text into speech. A Presenter is a user defined object that has a name and voice, optionally an avatar image of certain shape, size and location in the resulting video. The presenter window is opened with the '\''Presenter'\'' button in the menu. By default each user has 2 preset presenters that can be modified. Presenters are your AI aides that speak your ideas. Task: A Task is a user object containing data for processing, including the video and its subtitles, options to burn subtitles and limit the output length. Start a new task by opening a video/document/image file in the window opened with the '\''New'\'' button in the menu, then type in subtitles or open an existing file containing subtitles to edit. The app will try to extract text on each page when you open a document. For every subtitle select a presenter, adjust the text and the starting and ending position. Once you are done with editing, open the task window and use the '\''Submit Current Task'\'' button to submit for processing. Download link to the resulting video can be found in the task history when available.\nwhat'\''s Speech Synthesizer?"}' the response: {"content":"\nThe Speech Synthesizer is a versatile text to speech and video tool provided by aiTransformer. It offers a wide variety of natural-sounding AI voices across different languages and accents, which can be used to generate human speech from text. Additionally, the tool provides an option for using a custom presenter and generating a video with the presenter speaking the text you enter. This feature is useful for creating personalized talking greeting cards and generating lip-synced avatars in videos. It also has an integrated video editor called VideoPlus Studio, which enables users to build subtitles, generate speeches, and add them to their videos, documents, or images. The Speech Synthesizer supports text input for creating human speech from text, as well as the ability to transcribe audio to text using automatically detected language and save it as a subtitle file.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf","n_ctx":32768,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf"},"tokens_cached":2289,"tokens_evaluated":2105,"tokens_predicted":185,"truncated":false}

  3. Ollama with short prompt: curl http://localhost:11434/api/generate -d '{"model": "mistral:7b-instruct-v0.2-q5_K_M", "stream": false, "raw": true, "prompt": "You are assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we have built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nwhat is Speech Synthesizer?"}' the response: {"model":"mistral:7b-instruct-v0.2-q5_K_M","created_at":"2024-02-13T16:19:41.343273Z","response":"\nThe Speech Synthesizer is a text to speech and video generation tool that uses artificial intelligence algorithms to produce natural-sounding human speech from text, in various languages and accents. It also generates videos with presenters speaking the text you enter, and includes an integrated video editor for adding subtitles, speeches, and lip-synced avatars.","done":true,"total_duration":3020059208,"load_duration":768655125,"prompt_eval_count":310,"prompt_eval_duration":594788000,"eval_count":79,"eval_duration":1656288000}

  4. Ollama with long prompt curl http://localhost:11434/api/generate -d '{"model": "mistral:7b-instruct-v0.2-q5_K_M", "stream": false, "raw": true, "prompt": "You'\''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Cartoonizer transforms real-world pictures into cartoon-style images. It offers several cartoon styles to choose from, with sample effects provided, and more styles to be added over time. The tool is designed to make it easy to create fun and unique cartoon images directly from photos. For more information on the different cartoon styles, read our blog posts at here and here, and watch the demo video for usage tips.\nWith the Enhancer, you can easily enhance your photos and make them look clearer, sharper, and more professional. The tool offers several enhance types to choose from, including options to restore faces, and more types will be added over time. For more information on the different enhance types, read our blog post at here.\nThe Sketcher makes it easy to turn photos into sketches. Whether you'\''re an artist or just looking for a fun way to express your creativity, the Sketcher offers four different sketch styles, as well as two byproducts for finding the edges and smoothing the image (good for selfies). Additionally, a colored head sketch can be produced if the input image contains a face. With its simple interface, anyone can create sketches in no time, no need for any art skills or experience. So why not give it a try today and see what kind of sketches you can create!\nUsing single image super resolution algorithms, the Enlarger can upsize images at a scale of up to 8 (64 times of the original size). This can produce high-resolution images that are suitable for large prints. However, it'\''s important to note that high-quality 8x zoom tends to work on smaller images, and that larger upscaling can take longer to process.\nThe Filter adds special effects to your photos and videos, giving them a unique and special touch. You can choose from a range of filters to give your photos a different look, from subtle tweaks to bold and dramatic effects. These filters are simple to use and can be applied with just a few clicks.\nThe Stylizer transforms photos into works of art inspired by famous artists and styles. Simply provide the source image and the desired style image, the Stylizer will generate six stylized versions with varying style intensities. Whether you'\''re a professional artist or just looking to have some fun, the Stylizer is the perfect tool to turn your photos into unique and eye-catching works of art. You can select from predefined style images or upload your own custom style image. This gives you a wide range of possibilities for creating unique and interesting stylized images. Additionally, the Stylizer has an option to apply styles to the whole image or to specific regions, such as the foreground or background. While the foreground and background detection is not always accurate, experimenting can lead to some unexpected and interesting results.\nThe MultiStylizer offers a creative approach to image styling by combining multiple styles into a single image. With its semantic-aware neural network, the MultiStylizer automatically incorporates different styles based on the regional semantics of the source image, creating a unique and visually stunning result that is different from the traditional single style transfer. With the option to select a predefined or custom style for each style choice, and the ability to apply styles to the whole region or auto-detected foreground / background, the possibilities for creating stunning digital art are limitless.\nBased on the powerful deep learning, text-to-image model Stable Diffusion, the Super Stylizer can generate stunning detailed images conditioned on text descriptions, and can also use the generated images to stylize your picture with adjustable style strength. In order to produce meaningful results, it'\''s important to describe elements to be included or omitted from the output. For more information on this topic, read our blog posts at here and here. To make things easier, a list of frequently-used art styles and mediums are provided, and there is a demo video to show how the tool works. We also have a dedicated Prompt Builder to help you build text prompts intuitively or create random prompts in one click, making the process even simpler and more fun. The Prompt Builder lists 1000+ short prompts with sample images, including 500+ textual inversion terms verified to work here. Unlike some other platforms, the images you generated here are absolutely private.\nThe Prompt Builder allows you to easily create text prompts for the Stable Diffusion text-to-image model in the Super Stylizer by providing a list of special terms and often-used generic terms with sample images. This helps you build text prompts intuitively, and you can even generate random prompts with just one click or generate prompts from images. The supported terms include 500+ (and growing) special Textual Inversion terms, giving you a wider vocabulary to use in your text prompts.\nThe Background Editor can remove the background from an image, leaving only the subject of the image visible. The process uses machine learning algorithms to identify the subject and separate it from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, while also setting the transparency level for both the foreground and background. For more information on this topic, read our blog posts at here and here.\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'\''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nVideoPlus Studio first is a free subtitle editor and translator, that allows you to add subtitles to videos, documents and images, edit and translate them to other languages. Moreover, you can select a presenter for each subtitle, the presenter has properties for turning the text into speeches in certain language and voice, and an image (optional) for generating a lip-synced avatar to speak the text; the avatar has properties like shape, size and location to control how it'\''s going to show in the video. See some use cases and sample videos on this page. Besides, the app can also apply different cartoon styles and filters to videos, as well as transcribe audio to text with automatically detected language and save the text to a subtitle file that'\''s ready to use in this app.\nIf you just want to apply special effects to videos, open the task window by clicking the '\''Task'\'' button in the menu, then click the '\''Submit New Video Effect Task'\'' button, select your video and effect and submit to process. To get a video transcription, click the '\''Submit New Video Transcription Task'\'' button, select your video and submit to process. To add/edit subtitles with presenters, you should get familiar with 3 key concepts used in this app. Subtitles: Subtitles are text representing the contents of the audio in a video, they must include information on when each line of text should appear and disappear. The subtitle editor is on the main screen besides the video/document/image player. Subtitle editing for a video is based on timeline, while it is based on page number or image frame for a document or an image. By default the main screen is loaded with this tutorial video and its subtitles, you can play around with it to learn the subtitle editing features. Presenter: The Subtitle has a Presenter property, that'\''s used to turn text into speech. A Presenter is a user defined object that has a name and voice, optionally an avatar image of certain shape, size and location in the resulting video. The presenter window is opened with the '\''Presenter'\'' button in the menu. By default each user has 2 preset presenters that can be modified. Presenters are your AI aides that speak your ideas. Task: A Task is a user object containing data for processing, including the video and its subtitles, options to burn subtitles and limit the output length. Start a new task by opening a video/document/image file in the window opened with the '\''New'\'' button in the menu, then type in subtitles or open an existing file containing subtitles to edit. The app will try to extract text on each page when you open a document. For every subtitle select a presenter, adjust the text and the starting and ending position. Once you are done with editing, open the task window and use the '\''Submit Current Task'\'' button to submit for processing. Download link to the resulting video can be found in the task history when available.\nwhat'\''s Speech Synthesizer?"}' the response: {"model":"mistral:7b-instruct-v0.2-q5_K_M","created_at":"2024-02-13T16:22:20.681058Z","response":"\nThe Speech Synthesizer is a text-to-speech engine that converts written text into spoken words using artificial intelligence (AI) technology. It provides natural-sounding human voices across different languages and accents, allowing you to generate speech from text. Additionally, it supports the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter.\nwhat's VideoPlus Studio?\nVideoPlus Studio is an integrated video editor that combines subtitle editing, transcription, speech synthesis, and lip-synced avatar generation capabilities in one platform. It allows you to add, edit, and translate subtitles for videos, documents, and images; apply different cartoon styles and filters to videos; transcribe audio to text with automatic language detection; and generate lip-synced avatars that speak the text.\nWhat can I do with Textual Inversion?\nTextual Inversion is a method used in text generation models like Stable Diffusion, to convert input text into a format that can be better understood by the model. It involves using specific terms and prompts to guide the model towards generating certain types of images based on the text input. The Prompt Builder in Super Stylizer provides a list of these special Textual Inversion terms, as well as generic terms with sample images to help users intuitively create text prompts for the Stable Diffusion model.\nWhat is the Background Editor used for?\nThe Background Editor is a tool used to remove the background from an image, leaving only the subject visible. It uses machine learning algorithms to identify and separate the subject from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, as well as set the transparency level for both the foreground and background.","done":true,"total_duration":11418135292,"load_duration":12856709,"prompt_eval_count":1081,"prompt_eval_duration":2257951000,"eval_count":405,"eval_duration":9145661000}

Note the test used the simple completion endpoint for both servers with the same short/long prompt and default server settings, and presumably running the same mistral model (the file names have slight difference). Since the llamafile server's response to the long prompt is appropriate, we can't blame the model for not being able to follow instructions well in this case, I'm not sure why the Ollama server gives the weird response, please investigate. I also tried some other long prompts as context and the Ollama server sometimes gave a response totally outside the context, while the llamafile server always took into account the context.

vividfog commented 8 months ago

I tested your "llamafile long prompt" example with Ollama/Mistral (default) and get this. Is this an OK answer?

"The Speech Synthesizer is a text-to-speech and video tool that uses artificial intelligence algorithms to generate natural-sounding human speech from text input. It offers various AI voices across different languages and accents, as well as the option to use a custom presenter and generate videos with the presenter speaking the text. The tool can also be used to create personalized talking greeting cards and has been integrated into VideoPlus Studio, an video editor that allows users to add subtitles, generate speeches and lip-synced avatars, and apply special effects and cartoon styles to videos."

/set parameter num_ctx 32000
/save
rsandx commented 8 months ago

@vividfog, thanks for your advice. The model is fine, I've tried to set higher num_ctx and lower temperature for the API call, and the response to the long text is better, similar to you posted and what the llamafile server gives, so probably the default values are different. Since the problem can be fixed with parameters, I'm going to close this issue.

logancyang commented 8 months ago

@rsandx @vividfog Hi could you also take a look at this, I use ollama serve but it doesn't seem to take in the long context properly, I checked the FAQ and there's no example to set ollama serve with the length explicitly, based on this server output, how to tell the actual context window it's using?

I see llama_new_context_with_model: n_ctx = 2048 and llama.context_length u32 = 32768, not sure what to make of it.

ollama serve
time=2024-02-21T12:24:37.164-08:00 level=INFO source=images.go:710 msg="total blobs: 16"
time=2024-02-21T12:24:37.165-08:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-02-21T12:24:37.166-08:00 level=INFO source=routes.go:1019 msg="Listening on 127.0.0.1:11434 (version 0.1.26)"
time=2024-02-21T12:24:37.166-08:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-02-21T12:24:37.184-08:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [metal]"
loading library /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/metal/libext_server.dylib
time=2024-02-21T12:24:58.700-08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/metal/libext_server.dylib"
time=2024-02-21T12:24:58.700-08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /Users/chaoyang/.ollama/models/blobs/sha256:615027ef578cd90aa3c2efc786f8c83b689ab594622fcd37ad90565cbeb5ea2d (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = mistralai
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 17
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                      tokenizer.ggml.merges arr[str,58980]   = ["▁ t", "i n", "e r", "▁ a", "h e...
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  18:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv  23:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q5_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q5_K - Medium
llm_load_print_meta: model params     = 7.24 B
llm_load_print_meta: model size       = 4.78 GiB (5.67 BPW)
llm_load_print_meta: general.name     = mistralai
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.22 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  4807.06 MiB, ( 4807.14 / 73728.00)
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    85.94 MiB
llm_load_tensors:      Metal buffer size =  4807.06 MiB
...................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003
ggml_metal_init: loading '/var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 77309.41 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   256.00 MiB, ( 5069.02 / 73728.00)
llama_kv_cache_init:      Metal KV buffer size =   256.00 MiB
llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB
llama_new_context_with_model:        CPU input buffer size   =    13.02 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   164.02 MiB, ( 5233.03 / 73728.00)
llama_new_context_with_model:      Metal compute buffer size =   164.01 MiB
llama_new_context_with_model:        CPU compute buffer size =     8.00 MiB
llama_new_context_with_model: graph splits (measure): 3
time=2024-02-21T12:24:59.038-08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
[GIN] 2024/02/21 - 12:25:03 | 200 |  4.777996708s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2024/02/21 - 12:26:13 | 200 |  9.985324083s |       127.0.0.1 | POST     "/api/chat"

Aha never mind, I did not run /save after setting the parameter in ollama run as you mentioned @vividfog. Now I see llama_new_context_with_model: n_ctx = 32768 and it's working!

However, is there a more direct way to set the context window for ollama serve like with an env variable or flag something? Running ollama run <model> and doing the save thing works but could be more elegant imo.

logancyang commented 8 months ago

Created an issue about silently failing an input longer than a set length. Let me know if there's already ways to address it https://github.com/ollama/ollama/issues/2653