LLM not outputting text most of the time with Kobaldcpp or ollama back end.

[reniv1]

I am having trouble getting usable if any text to come out of the response pin, the question pin comes out as expected tho. The llm back end receives the question, but nothing is coming out. I am using the If chat prompt node. My workflow is just a if chat prompt connected to a show text in the 4 pins. I am on a windows os computer.

I have tried changing the stop, using <|end_of_text|>, < /s>, . , and <|eot_id|>. I also changed the assistant to MKR, and the model to llama3_if_ai_sdpromptmkr_q4km. I have tried a few other models also, mostly llama 3 or 2 models. Besides that I left everything default besides the port numbers. Is there anyway to just get the raw text out? I tried flipping the text cleanup, but nothing happened. Also changed assistant to none. I have nothing going into the image or context pin, just a command in the prompt box. Thanks for your help.

Below is the output from Koboldcpp back end terminal, first one being MKR with the llama 3_if_ai model, second being none as the assistant and a llama 3 model. I had similar results with ollama, but it wasn't showing anything in the terminal.

Input: {"prompt": "System: {'instruction': ' You are a prompt maker. Create a high-quality, coherent, and concise prompts based on the given subject, following the provided guidelines and format.', 'rules': ['Break keywords by commas', 'Focus solely on visual elements; avoid art commentaries or intentions', 'Construct prompt with subject, scene, and background components', 'Limit to 7 keywords per component', 'Include all subject keywords verbatim as main focus', 'Be varied and creative in descriptions', 'Keep prompt under 100 words', 'Do not enumerate or enunciate components', 'Do not include additional information beyond prompt'], 'examples': [{'input': 'Demon Hunter, Cyber City', 'output': 'A Demon Hunter, standing, lone figure, glowing eyes, deep purple light, cybernetic exoskeleton, sleek, metallic, glowing blue accents, energy weapons, fighting Demon, grotesque creature, twisted metal, glowing red eyes, sharp claws, in Cyber City, towering structures, shrouded haze, shimmering energy'}]}\nUser: I want you to describe a photo of a forest in detail.\nUser: I want you to describe a photo of a forest in detail.", "max_length": 128, "temperature": 0.7, "top_k": 40, "top_p": 0.2, "rep_pen": 1.1, "stop_sequence": ["\n\n\n\n\n", "<|end_of_text|>"]}

Processing Prompt (29 / 29 tokens) Generating (128 / 128 tokens) CtxLimit: 394/8192, Process:0.00s (0.1ms/T = 7250.00T/s), Generate:1.88s (14.7ms/T = 68.01T/s), Total:1.89s (67.87T/s) Output:

System: Serene, mystical forest landscape, dense foliage, towering trees, vibrant green leaves, soft sunlight filtering through canopy above, dappled shadows beneath, moss-covered rocks, hidden streams, misty veil rising from valley floor, ethereal atmosphere, dreamlike quality, captivating composition, rich colors, depth of field, clarity, vividness, serenity, enchantment.

Break keywords by commas: serene, mystical, forest, landscape, dense, foliage, towering, trees, vibrant, green, leaves, soft, sunlight, filtering, canopy, above, dappled, shadows, beneath, moss-covered, rocks,

Input: {"prompt": "System: \nUser: I want you to describe a photo of a forest in detail.\nUser: I want you to describe a photo of a forest in detail.", "max_length": 8000, "temperature": 0.7, "top_k": 40, "top_p": 0.2, "rep_pen": 1.1, "stop_sequence": ["\n\n\n\n\n", "<|end_of_text|>"]}

Processing Prompt (31 / 31 tokens) Generating (228 / 8000 tokens) (EOS token triggered!) (Special Stop Token Triggered! ID:128001) CtxLimit: 262/8192, Process:0.00s (0.1ms/T = 10333.33T/s), Generate:3.92s (0.5ms/T = 2041.86T/s), Total:3.92s (2040.30T/s) Output:

Response: The photo depicts a dense and lush forest, with tall trees that stretch up towards the sky, their leaves rustling gently in the breeze. The sunlight filters through the canopy above, casting dappled patterns of light and shadow on the forest floor below. Various shades of green dominate the scene, from the deep emerald of the tree trunks to the vibrant foliage of the branches and leaves. In the distance, one can see the silhouette of a winding river, its surface reflecting the colors of the surrounding landscape.

In addition to the main focus on the trees, there are also smaller details scattered throughout the image. A carpet of moss covers much of the forest floor, providing a soft and damp contrast to the rough bark of the tree trunks. Some fallen logs lie strewn about, adding texture and depth to the overall composition. Birds and insects flit between the branches, further bringing life to this already thriving ecosystem.

Overall, the photo captures the beauty and tranquility of a forest at its peak, showcasing both the grandeur of nature as well as the intricate details that make it so unique.

[merc4derp]

Seconding this. Been trying to get koboldcpp to work for a while now with this node. Can't get anything usable out of it.

Only the chat prompt node has a kobold backend as option and none of the outputs provided by the node actually carry the messages coming from koboldcpp despite my best efforts.

Something seems broken in the api implementation or better instructions needed for how this is supposed to work.

[if-ai]

Sorry I am fixing it now I just got home I had an issue with the images not sure what is happening it was working before I am working on it

[if-ai]

Koboldcpp says that they have support for image messages in their API but I tried everything with both the open ai endpoint and the native endpoint and it does not pick the image maybe they have support images but only with as particular model. for now only text seems to be working please try again after updating

[reniv1]

Thanks for your help, It works well. The text was what i planned on using for just adding a bit of randomness to prompts.

[if-ai]

Thank you. I hope they improve the vision model support api. Is the same with oobabooga they only support some old vision models which restricts what I could do.