Run on koboldcpp... - Githubissues

I copy the example python and run it. The only thing I change is comment the openai API and release local server setting:

os.environ["OPENAI_API_BASE"] = 'https://xxxxxxx-5000.proxy.runpod.net/v1'
os.environ["OPENAI_MODEL_NAME"] ='openhermes'  # Adjust based on available model
os.environ["OPENAI_API_KEY"] ='sk-111111111111111111111111111111111111111111111111'

it is a koboldcpp server, run with meta llama 3 70B Q4XS gguf model.

wget https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/resolve/main/Meta-Llama-3-70B-Instruct.IQ4_XS.gguf
mv *.gguf model.gguf
./koboldcpp --model model.gguf --noshift --smartcontext --host 0.0.0.0 --port 5000 --contextsize 16384 --blasbatchsize 2048 --usecublas 0 0 --gpulayers 999 --multiuser

After I run the program, it get the answer:

Input: {"messages": [{"role": "user", "content": "You are Tech Content Strategist. You are a renowned Content Strategist, known for your insightful and engaging articles.\n  You transform complex concepts into compelling narratives.\nYour personal goal is: Craft compelling content on tech advancements\nYou ONLY have access to the following tools, and should NEVER make up tools that are not listed here:\n\nDelegate work to co-worker: Delegate work to co-worker(coworker: str, task: str, context: str) - Delegate a specific task to one of the following co-workers: [Senior Research Analyst]\nThe input to this tool should be the coworker, the task you want them to do, and ALL necessary context to exectue the task, they know nothing about the task, so share absolute everything you know, don't reference things but instead explain them.\nAsk question to co-worker: Ask question to co-worker(coworker: str, question: str, context: str) - Ask a specific question to one of the following co-workers: [Senior Research Analyst]\nThe input to this tool should be the coworker, the question you have for them, and ALL necessary context to ask the question properly, they know nothing about the question, so share absolute everything you know, don't reference things but instead explain them.\n\nUse the following format:\n\nThought: you should always think about what to do\nAction: the action to take, only one name of [Delegate work to co-worker, Ask question to co-worker], just the name, exactly as it's written.\nAction Input: the input to the action, just a simple a python dictionary using \" to wrap keys and values.\nObservation: the result of the action\n\nOnce all necessary information is gathered:\n\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nCurrent Task: Using the insights provided, develop an engaging blog\n  post that highlights the most significant AI advancements.\n  Your post should be informative yet accessible, catering to a tech-savvy audience.\n  Make it sound cool, avoid complex words so it doesn't sound like AI.\n\nThis is the expect criteria for your final answer: Full blog post of at least 4 paragraphs \n you MUST return the actual complete content as the final answer, not a summary.\n\nThis is the context you're working with:\n**Comprehensive Analysis of AI Advancements in 2024**\n\n**Key Trends:**\n\n* **Explainability and Transparency**: AI models will prioritize interpretability, enabling users to understand decision-making processes and ensure accountability.\n* **Edge AI**: With the proliferation of IoT devices, AI will be increasingly deployed at the edge to reduce latency and enhance real-time processing capabilities.\n* **Autonomous Systems**: AI-powered autonomous systems will become more prevalent\n\nBegin! This is VERY important to you, use the tools available and give your best Final Answer, your job depends on it!\n\nThought: \nThought: I need to get more insights on the AI advancements to craft a compelling blog post. \n\nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Explainability and Transparency trend?\", \"context\": \"I want to understand how AI models will prioritize interpretability and ensure accountability. What are some real-world examples of this trend?\"}\n\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistake. Here is the corrected action:\n\nThought: I need to get more insights on the AI advancements to craft a compelling blog post. \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Explainability and Transparency trend?\", \"context\": \"I want to understand how AI models will prioritize interpretability and ensure accountability. What are some real-world examples of this trend?\"}\n\nObservation: Explainability and Transparency (XAI) is a growing trend in AI development, focusing on creating models that provide clear insights into their decision-making processes. This is crucial for building trust, ensuring accountability, and identifying biases in AI systems. \n\nIn real-world applications, Explainability and Transparency are essential for high-stakes domains like healthcare, finance, and transportation, where AI-driven decisions can have significant consequences. For instance, in healthcare\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend. \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistake again. Here is the corrected action:\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistake. Here is the corrected action:\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistake. Here is the corrected action:\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistake. Here is the corrected action:\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I apologize for the mistake. Let me correct the action input. \n\nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I apologize for the mistake. Let me correct the action input. \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I need to reformat the Action Input to a valid key-value dictionary.\n\nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I will retry the action input with correct formatting.\nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I need to reformat the Action Input to a valid key-value dictionary.\nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I will retry the action input with correct formatting.\nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nThought: I will retry the action input with correct formatting.assistant\n\nThought: I will retry the action input with correct formatting.\n\nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"}\n\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nLet me try again. Here is the corrected action input:\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistake. Here is the corrected action:\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.\"\nObservation: Error: the Action Input is not a valid key, value dictionary.\nIt seems like there's an issue with the Action Input formatting. Let's try again!\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {\"coworker\": \"Senior Research Analyst\", \"question\": \"Can you provide more details on the Edge AI trend?\", \"context\": \"I want to understand how AI will be deployed at the edge and what kind of real-time\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the mistakes earlier. It seems like there's an issue with the Action Input formatting. Let's try again!\n\nThought: I have some information on Explainability and Transparency, but I need more insights on the Edge AI trend.  \nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', question: 'Can you provide more details on the Edge AI trend?', context: 'I want to understand how AI will be deployed at the edge and\nObservation: Error: the Action Input is not a valid key, value dictionary.\nI apologize for the repeated mistakes. Let me start again from the beginning.\n\nThought: I need to get more insights on the AI advancements to craft a compelling blog post.\nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'}\nObserv\nObservation: Error: the Action Input is not a valid key, value dictionary.\nLet me start again from the beginning.\nThought: I need to get more insights on the AI advancements to craft a compelling blog post. \nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nIt seems like there's an issue with the Action Input formatting. I'll start again from the beginning.\n\nThought: I need to get more insights on the AI advancements to craft a compelling blog post.\nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities\nObservation: Error: the Action Input is not a valid key, value dictionary.\nIt seems like there's an issue with the Action Input formatting. I'll start again from the beginning.\nAction: Ask question to co-worker\nAction Input: {'coworker': 'Senior Research Analyst', 'question': 'Can you provide more details on the Edge AI trend?', 'context': 'I want to understand how AI will be deployed at the edge and what kind of real-time processing capabilities it will bring.'}\nObservation\nObservation: Error: the Action Input is not a valid key, value dictionary.\nTool won't be use because it's time to give your final answer. Don't use tools and just your absolute BEST Final answer.\nObservation: Tool won't be use because it's time to give your final answer. Don't use tools and just your absolute BEST Final answer.\n"}], "model": "openhermes", "n": 1, "stop": ["\nObservation"], "stream": true, "temperature": 0.7}

Processing Prompt [BLAS] (172 / 172 tokens)
Generating (100 / 100 tokens)
CtxLimit: 3480/16384, Process:0.67s (3.9ms/T = 255.57T/s), Generate:8.44s (84.4ms/T = 11.85T/s), Total:9.11s (10.97T/s)
Output: Thought: I now know the final answer.

Final Answer:

**Comprehensive Analysis of AI Advancements in 2024**

As we step into 2024, the world of Artificial Intelligence (AI) is poised to witness significant advancements that will revolutionize the way we live and work. From Explainability and Transparency to Edge AI and Autonomous Systems, these emerging trends are set to transform industries and reshape our future.

**Explainability and Transparency: The Dawn of Trustworthy AI**

In 202

yes, it stop after it generates 100 tokens every time. Clearly, the program didn't submit max_token parameter to koboldcpp server, so it didn't generate more than 100 tokens. Koboldcpp didn't have initialize parameter to config the response token in its initial setting, it is submitted in request, and I didn't find where I can change the code to add the max_token parameter in the request.

crewAIInc / crewAI

Run on koboldcpp... #513