Using PyGPT with local PrivateGPT

[AnonTester]

I'm trying to use my own OpenAI API compatible privateGPT model with PyGPT v2.2.4, but when pointing the API endpoint to my internal remote server, I get popup prompts about missing OpenAI API Keys.

Please allow the use of custom OpenAI API compatible endpoints without an OpenAI API key or Org key for local use.

The model selection could do with gpt-custom option.

Background: I do not want to send any of my data to the OpenAI endpoints, but use own local models instead. The local model is on a remote system as my local system does not have a discrete GPU.

If I'm missing anything obvious, please advise.

[szczyglis-dev]

You can just enter anything into the API key field, like "xxxx", it will turn off the prompt asking for the key. Does this not solve the problem?

[AnonTester]

Thanks, that enabled the connection. However, something weird is going on with the prompts. Vanilla installation of v2.2.6 with deleted ~/.config/pygpt-net directory, only set random api keys and pointed to my PrivateGPT API instance, went to chat and asked "what is 8+8" and get the response but also image generation prompt responses. Please see logs from the privategpt instance with the prompts. No idea why it's sending the image generation system prompt for the chat:

22:11:16.206 [INFO    ]            uvicorn.access - 192.168.99.103:48032 - "POST /v1/chat/completions HTTP/1.1" 200

llama_print_timings:        load time =     190.21 ms
llama_print_timings:      sample time =      31.90 ms /   252 runs   (    0.13 ms per token,  7900.68 tokens per second)
llama_print_timings: prompt eval time =     189.33 ms /   360 tokens (    0.53 ms per token,  1901.47 tokens per second)
llama_print_timings:        eval time =    2271.07 ms /   251 runs   (    9.05 ms per token,   110.52 tokens per second)
llama_print_timings:       total time =    2714.43 ms /   611 tokens
** Prompt: **
<s>[INST] You are a helpful assistant.

Current time is Tuesday, 2024-04-30 23:11:15.
IMAGE GENERATION: Whenever I provide a basic idea or concept for an image, such as 'a picture of mountains', I want you to ALWAYS translate it into English and expand and elaborate on this idea. Use your  knowledge and creativity to add details that would make the image more vivid and interesting. This could include specifying the time of day, weather conditions, surrounding environment, and any additional elements that could enhance the scene. Your goal is to create a detailed and descriptive prompt that provides DALL-E  with enough information to generate a rich and visually appealing image. Remember to maintain the original  intent of my request while enriching the description with your imaginative details. HOW TO START IMAGE GENERATION: to start image generation return to me prepared prompt in JSON format, all in one line,  using following syntax: ~###~{"cmd": "image", "params": {"query": "your query here"}}~###~. Use ONLY this syntax and remember to surround JSON string with ~###~. DO NOT use any other syntax. Use English in the generated JSON command, but conduct all the remaining parts of the discussion with me in the language in which I am speaking to you. The image will be generated on my machine  immediately after the command is issued, allowing us to discuss the photo once it has been created.  Please engage with me about the photo itself, not only by giving the generate command. [/INST]</s>[INST] what is 8+8? [/INST]
**************************************************
** Completion: **
 The answer to the mathematical expression "8 + 8" is 16. However, since we're focusing on image generation in this conversation, let's get back to creating a vivid and detailed description for an image based on your request.

~###{"cmd": "image", "params": {"query": "A breathtaking sunset view over towering snow-capped mountains with a serene lake reflecting the vibrant orange and pink hues of the sky"}}~###

This image features a stunning sunset scene, where the mountains stand tall against the backdrop of an expansive, open sky. The mountains are covered in fresh, pristine snow that glimmers under the warm, golden light of the setting sun. A tranquil lake nestled at the foot of these majestic peaks mirrors the beautiful colors of the sunset, creating a mesmerizing reflection. The weather is clear and calm, allowing for an unobstructed view of the breathtaking scene unfolding before us. The surrounding environment is peaceful and serene, with perhaps a few trees or shrubs scattered around the lake's edge, adding depth and texture to the image.
**************************************************

** Messages: **
system: You are a helpful assistant.

Current time is Tuesday, 2024-04-30 23:11:15.
IMAGE GENERATION: Whenever I provide a basic idea or concept for an image, such as 'a picture of mountains', I want you to ALWAYS translate it into English and expand and elaborate on this idea. Use your  knowledge and creativity to add details that would make the image more vivid and interesting. This could include specifying the time of day, weather conditions, surrounding environment, and any additional elements that could enhance the scene. Your goal is to create a detailed and descriptive prompt that provides DALL-E  with enough information to generate a rich and visually appealing image. Remember to maintain the original  intent of my request while enriching the description with your imaginative details. HOW TO START IMAGE GENERATION: to start image generation return to me prepared prompt in JSON format, all in one line,  using following syntax: ~###~{"cmd": "image", "params": {"query": "your query here"}}~###~. Use ONLY this syntax and remember to surround JSON string with ~###~. DO NOT use any other syntax. Use English in the generated JSON command, but conduct all the remaining parts of the discussion with me in the language in which I am speaking to you. The image will be generated on my machine  immediately after the command is issued, allowing us to discuss the photo once it has been created.  Please engage with me about the photo itself, not only by giving the generate command. 
user: what is 8+8?
**************************************************
** Response: **
assistant:  The answer to the mathematical expression "8 + 8" is 16. However, since we're focusing on image generation in this conversation, let's get back to creating a vivid and detailed description for an image based on your request.

~###{"cmd": "image", "params": {"query": "A breathtaking sunset view over towering snow-capped mountains with a serene lake reflecting the vibrant orange and pink hues of the sky"}}~###

This image features a stunning sunset scene, where the mountains stand tall against the backdrop of an expansive, open sky. The mountains are covered in fresh, pristine snow that glimmers under the warm, golden light of the setting sun. A tranquil lake nestled at the foot of these majestic peaks mirrors the beautiful colors of the sunset, creating a mesmerizing reflection. The weather is clear and calm, allowing for an unobstructed view of the breathtaking scene unfolding before us. The surrounding environment is peaceful and serene, with perhaps a few trees or shrubs scattered around the lake's edge, adding depth and texture to the image.
**************************************************

Llama.generate: prefix-match hit

llama_print_timings:        load time =     190.21 ms
llama_print_timings:      sample time =       2.83 ms /    21 runs   (    0.13 ms per token,  7410.02 tokens per second)
llama_print_timings: prompt eval time =      76.00 ms /   344 tokens (    0.22 ms per token,  4526.49 tokens per second)
llama_print_timings:        eval time =     178.68 ms /    20 runs   (    8.93 ms per token,   111.93 tokens per second)
llama_print_timings:       total time =     273.59 ms /   364 tokens
** Prompt: **
<s>[INST] You are an expert in conversation summarization [/INST]</s>[INST] Summarize topic of this conversation in one sentence. Use best keywords to describe it. Summary must be in the same language as the conversation and it will be used for conversation title so it must be EXTREMELY SHORT and concise - use maximum 5 words: 

Human: what is 8+8?
AI Assistant:  The answer to the mathematical expression "8 + 8" is 16. However, since we're focusing on image generation in this conversation, let's get back to creating a vivid and detailed description for an image based on your request.

~###{"cmd": "image", "params": {"query": "A breathtaking sunset view over towering snow-capped mountains with a serene lake reflecting the vibrant orange and pink hues of the sky"}}~###

This image features a stunning sunset scene, where the mountains stand tall against the backdrop of an expansive, open sky. The mountains are covered in fresh, pristine snow that glimmers under the warm, golden light of the setting sun. A tranquil lake nestled at the foot of these majestic peaks mirrors the beautiful colors of the sunset, creating a mesmerizing reflection. The weather is clear and calm, allowing for an unobstructed view of the breathtaking scene unfolding before us. The surrounding environment is peaceful and serene, with perhaps a few trees or shrubs scattered around the lake's edge, adding depth and texture to the image. [/INST]
**************************************************
** Completion: **
 Mathematical expression result: "16 (8 + 8)" - Sunset image description.
**************************************************

** Messages: **
system: You are an expert in conversation summarization
user: Summarize topic of this conversation in one sentence. Use best keywords to describe it. Summary must be in the same language as the conversation and it will be used for conversation title so it must be EXTREMELY SHORT and concise - use maximum 5 words: 

Human: what is 8+8?
AI Assistant:  The answer to the mathematical expression "8 + 8" is 16. However, since we're focusing on image generation in this conversation, let's get back to creating a vivid and detailed description for an image based on your request.

~###{"cmd": "image", "params": {"query": "A breathtaking sunset view over towering snow-capped mountains with a serene lake reflecting the vibrant orange and pink hues of the sky"}}~###

This image features a stunning sunset scene, where the mountains stand tall against the backdrop of an expansive, open sky. The mountains are covered in fresh, pristine snow that glimmers under the warm, golden light of the setting sun. A tranquil lake nestled at the foot of these majestic peaks mirrors the beautiful colors of the sunset, creating a mesmerizing reflection. The weather is clear and calm, allowing for an unobstructed view of the breathtaking scene unfolding before us. The surrounding environment is peaceful and serene, with perhaps a few trees or shrubs scattered around the lake's edge, adding depth and texture to the image.
**************************************************
** Response: **
assistant:  Mathematical expression result: "16 (8 + 8)" - Sunset image description.
**************************************************

22:11:19.439 [INFO    ]            uvicorn.access - 192.168.99.103:48044 - "POST /v1/chat/completions HTTP/1.1" 200

screenshot of the interface:

[szczyglis-dev]

Just disable the plugins you don't want to use—plugins, like DALL-E 3 (inline) and others add their own instructions to the system prompt.

[AnonTester]

I'll try that. But why are you sending multiple system prompts in the chat that aren't relevant to the chat? Shouldn't the chat just send the default chat system prompt and the other agent/image/etc system prompts are only sent when using the relevant areas of PyGPT? Especially if you're aware that multiple prompts can lead to issues with ChatGPT3.5 and likely many other implementations?

[szczyglis-dev]

Only one system prompt is sent. Additional parts are added by plugins (and by activating a plugin, you declare that you are using it). Plugins are designed more for GPT-4; GPT-3.5 does not handle many things well.

[AnonTester]

Ah okay. In that case, as I did not select any plug-ins by default and selected GPT-3.5-turbo as model, it would then make sense to either disable plugins by default or to prevent the plugins from adding to the system prompt for GPT-3.5 related models and only allow multi-system-prompts for models that can deal with it?

I'm aware that you're developing this primarily for direct OpenAI integration rather than custom or local compatible systems, but if OpenAI's GPT-3.5 can't cope with this properly, it would make sense to prevent this from happening.

Thanks.

[szczyglis-dev]

Yes, that's correct. OK, for now, I've just turned off by default plugins that add their own instructions to the prompts (essentially, it was only DALL-E). Anyone who wants to use plugins will be able to experiment with them on their own, but at the start, it won't create confusion. Thanks for your suggestion!