BarfingLemurs
commented
1 year ago what is that extension? do you mean "openai"?
Closed Simplegram closed 1 year ago
BarfingLemurs
commented
1 year ago what is that extension? do you mean "openai"?
Simplegram
commented
1 year ago what is that extension? do you mean "openai"?
Oh yeah sorry, it should be openai.
oobabooga
commented
1 year ago @matatonic
matatonic
commented
1 year ago @Simplegram Hi, this looks like it's a case of gpt-4/3.5 just being better than the current crop of local models. When asked to output only json, they do it, when you ask vicuna... maybe sometimes and maybe not all there. I mention this in the Docs for the openai extension: https://github.com/matatonic/text-generation-webui/blob/main/extensions/openai/README.md#applications .
Just to be sure though, if you set the environment variable OPENEDAI_DEBUG=1 before you run server.py it will print the exact output returned from the API. I guess you'll see something there like Auto-GPT asking for a very specific json format (command:, thoughts:, etc) and then vicuna/etc replying something like "Sure here you go: ....
I've personally had slightly better luck with Alpaca (original) which seems less chatty and generally more machine like, but it isn't enough better to be useful with Auto-GPT.
As another user has pointed out in the langchain github (which has the exact same kinds of issues), Microsoft's new 'guidance' library may be the answer here, and I'm looking into that.
Simplegram
commented
1 year ago @Simplegram Hi, this looks like it's a case of gpt-4/3.5 just being better than the current crop of local models. When asked to output only json, they do it, when you ask vicuna... maybe sometimes and maybe not all there. I mention this in the Docs for the openai extension: https://github.com/matatonic/text-generation-webui/blob/main/extensions/openai/README.md#applications .
Just to be sure though, if you set the environment variable OPENEDAI_DEBUG=1 before you run server.py it will print the exact output returned from the API. I guess you'll see something there like Auto-GPT asking for a very specific json format (command:, thoughts:, etc) and then vicuna/etc replying something like "Sure here you go: .... " which wont work for Auto-GPT.
I've personally had slightly better luck with Alpaca (original) which seems less chatty and generally more machine like, but it isn't enough better to be useful with Auto-GPT.
As another user has pointed out in the langchain github (which has the exact same kinds of issues), Microsoft's new 'guidance' library may be the answer here, and I'm looking into that.
Hey, thanks a lot for your reply. Here's the debug log using TheBloke_wizard-mega-13B-GPTQ model , maybe you'll find this useful.
Auto-GPT log
All packages are installed.
NEWS: Welcome to Auto-GPT!
NEWS: Below you'll find the latest Auto-GPT News and updates regarding features!
NEWS: If you don't wish to see this message, you can run Auto-GPT with the --skip-news flag.
NEWS:
NEWS: Website and Documentation Site 📰📖
NEWS: Check out https://agpt.co, the official news & updates site for Auto-GPT!
NEWS: The documentation also has a place here, at https://docs.agpt.co
NEWS:
NEWS: For contributors 👷🏼
NEWS: Since releasing v0.3.0, we are working on re-architecting the Auto-GPT core to make
NEWS: it more extensible and to make room for structural performance-oriented R&D.
NEWS: In the meantime, we have less time to process incoming pull requests and issues,
NEWS: so we focus on high-value contributions:
NEWS: * significant bugfixes
NEWS: major* improvements to existing functionality and/or docs (so no single-typo fixes)
NEWS: * contributions that help us with re-architecture and other roadmapped items
NEWS: We have to be somewhat selective in order to keep making progress, but this does not
NEWS: mean you can't contribute. Check out the contribution guide on our wiki:
NEWS: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Contributing
NEWS:
NEWS: 🚀 v0.3.1 Release 🚀
NEWS: Over a week and 47 pull requests have passed since v0.3.0, and we are happy to announce
NEWS: the release of v0.3.1!
NEWS:
NEWS: Highlights and notable changes since v0.2.2:
NEWS:
NEWS: Changes to Docker configuration 🐋
NEWS: The workdir has been changed from /home/appuser to /app*.
NEWS: Be sure to update any volume mounts accordingly!
NEWS: * Docker-compose 1.29.0 is now required.
NEWS:
NEWS: Logging 🧾
NEWS: * Log functionality has been improved for better understanding
NEWS: and easier summarization.
NEWS: * All LLM interactions are now logged to logs/DEBUG, to help with
NEWS: debugging and development.
NEWS:
NEWS: Other
NEWS: * Edge browser is now supported by the `browse_website` command.
NEWS: * Sets of commands can now be disabled using DISABLED_COMMAND_CATEGORIES in .env.
NEWS:
NEWS: ⚠️ Command `send_tweet` is DEPRECATED, and will be removed in v0.4.0 ⚠️
NEWS: Twitter functionality (and more) is now covered by plugins, see [Plugin support 🔌]
NEWS:
NEWS: Plugin support 🔌
NEWS: Auto-GPT now has support for plugins! With plugins, you can extend Auto-GPT's abilities,
NEWS: adding support for third-party services and more.
NEWS: See https://github.com/Significant-Gravitas/Auto-GPT-Plugins for instructions and
NEWS: available plugins. Specific plugins can be allowlisted/denylisted in .env.
NEWS:
WARNING: Plugin AutoGPTBaiduSearch found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTBingSearch found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTBluesky found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTEmailPlugin found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTNewsSearch found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin PlannerPlugin found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTRandomValues found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTSceneXPlugin found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTTelegram found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTTwitter found. But not in the allowlist... Load? (y/n): n
WARNING: Plugin AutoGPTWikipediaSearch found. But not in the allowlist... Load? (y/n): n
Plugins found: 2
--------------------
autogpt-api-tools: 0.1.0 - Allow AutoGPT to make API calls to outside services.
Auto-GPT-Space-Plugin: 0.1.3 - Auto-GPT Space Plugin: Get the number of astronauts in space in real-time..
Welcome back! Would you like me to return to being DataFinderGPT?
Asking user via keyboard...
Continue with the last settings?
Name: DataFinderGPT
Role: An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently.
Goals: []
API Budget: infinite
Continue (y/n): y
NOTE:All files/directories created by this agent can be found inside its workspace at: E:\Projects\cpy\llms\auto-gpt\Auto-GPT\autogpt\auto_gpt_workspace
DataFinderGPT has been created with the following details:
Name: DataFinderGPT
Role: An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently.
Goals:
Using memory of type: LocalCache
Using Browser: chrome
Error: The following AI output couldn't be converted to a JSON:
</s>
NEXT ACTION: COMMAND = None ARGUMENTS = None
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands, 'n' to exit program, or enter feedback for DataFinderGPT...
Asking user via keyboard...
Input:Oobabooga Text Generation log
Host: 127.0.0.1:5001
User-Agent: OpenAI/v1 PythonBindings/0.27.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.2", "httplib": "requests", "lang": "python", "lang_version": "3.11.0", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
Authorization: Bearer dummy
Content-Type: application/json
Content-Length: 4076
{'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': 'You are DataFinderGPT, An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently.\nYour decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.\n\nGOALS:\n\n\n\nConstraints:\n1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.\n2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.\n3. No user assistance\n4. Exclusively use the commands listed below e.g. command_name\n\nCommands:\n1. analyze_code: Analyze Code, args: "code": "<full_code_string>"\n2. execute_python_file: Execute Python File, args: "filename": "<filename>"\n3. append_to_file: Append to file, args: "filename": "<filename>", "text": "<text>"\n4. delete_file: Delete file, args: "filename": "<filename>"\n5. list_files: List Files in Directory, args: "directory": "<directory>"\n6. read_file: Read file, args: "filename": "<filename>"\n7. write_to_file: Write to file, args: "filename": "<filename>", "text": "<text>"\n8. google: Google Search, args: "query": "<query>"\n9. improve_code: Get Improved Code, args: "suggestions": "<list_of_suggestions>", "code": "<full_code_string>"\n10. send_tweet: Send Tweet, args: "tweet_text": "<tweet_text>"\n11. browse_website: Browse Website, args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"\n12. write_tests: Write Tests, args: "code": "<full_code_string>", "focus": "<list_of_focus_areas>"\n13. delete_agent: Delete GPT Agent, args: "key": "<key>"\n14. get_hyperlinks: Get text summary, args: "url": "<url>"\n15. get_text_summary: Get text summary, args: "url": "<url>", "question": "<question>"\n16. list_agents: List GPT Agents, args: () -> str\n17. message_agent: Message GPT Agent, args: "key": "<key>", "message": "<message>"\n18. start_agent: Start GPT Agent, args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"\n19. task_complete: Task Complete (Shutdown), args: "reason": "<reason>"\n20. make_api_call: "Make an API call", args: "host": "<host>", "endpoint": "<endpoint>", "method": "<method>", "query_params": "<query_params>", "body": "<body>", "headers": "<headers>"\n21. Get number of astronauts: "get_num_astronauts", args: \n\nResources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\n5. Write all code to a file.\n\nYou should only respond in JSON format as described below \nResponse Format: \n{\n "thoughts": {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n },\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n }\n} \nEnsure the response can be parsed by Python json.loads'}, {'role': 'system', 'content': 'The current time and date is Sat May 20 07:53:08 2023'}, {'role': 'user', 'content': 'Determine which next command to use, and respond using the format specified above:'}], 'temperature': 0.0, 'max_tokens': 2576}
127.0.0.1 - - [20/May/2023 07:53:08] "POST /v1/chat/completions HTTP/1.1" 200 -
{'prompt': 'system: You are DataFinderGPT, An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently.\nYour decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.\n\nGOALS:\n\n\n\nConstraints:\n1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.\n2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.\n3. No user assistance\n4. Exclusively use the commands listed below e.g. command_name\n\nCommands:\n1. analyze_code: Analyze Code, args: "code": "<full_code_string>"\n2. execute_python_file: Execute Python File, args: "filename": "<filename>"\n3. append_to_file: Append to file, args: "filename": "<filename>", "text": "<text>"\n4. delete_file: Delete file, args: "filename": "<filename>"\n5. list_files: List Files in Directory, args: "directory": "<directory>"\n6. read_file: Read file, args: "filename": "<filename>"\n7. write_to_file: Write to file, args: "filename": "<filename>", "text": "<text>"\n8. google: Google Search, args: "query": "<query>"\n9. improve_code: Get Improved Code, args: "suggestions": "<list_of_suggestions>", "code": "<full_code_string>"\n10. send_tweet: Send Tweet, args: "tweet_text": "<tweet_text>"\n11. browse_website: Browse Website, args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"\n12. write_tests: Write Tests, args: "code": "<full_code_string>", "focus": "<list_of_focus_areas>"\n13. delete_agent: Delete GPT Agent, args: "key": "<key>"\n14. get_hyperlinks: Get text summary, args: "url": "<url>"\n15. get_text_summary: Get text summary, args: "url": "<url>", "question": "<question>"\n16. list_agents: List GPT Agents, args: () -> str\n17. message_agent: Message GPT Agent, args: "key": "<key>", "message": "<message>"\n18. start_agent: Start GPT Agent, args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"\n19. task_complete: Task Complete (Shutdown), args: "reason": "<reason>"\n20. make_api_call: "Make an API call", args: "host": "<host>", "endpoint": "<endpoint>", "method": "<method>", "query_params": "<query_params>", "body": "<body>", "headers": "<headers>"\n21. Get number of astronauts: "get_num_astronauts", args: \n\nResources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\n5. Write all code to a file.\n\nYou should only respond in JSON format as described below \nResponse Format: \n{\n "thoughts": {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n },\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n }\n} \nEnsure the response can be parsed by Python json.loadsThe current time and date is Sat May 20 07:53:08 2023\n\nuser: Determine which next command to use, and respond using the format specified above:\nassistant: ', 'req_params': {'max_new_tokens': 2576, 'temperature': 0.001, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': False, 'echo': False, 'seed': -1, 'truncation_length': 65000, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1, 'early_stopping': False, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}, 'stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}
Output generated in 3.94 seconds (1.27 tokens/s, 5 tokens, context 1111, seed 412636788)
{'response': '\u200b</s>'}
Host: 127.0.0.1:5001
User-Agent: OpenAI/v1 PythonBindings/0.27.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.2", "httplib": "requests", "lang": "python", "lang_version": "3.11.0", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
Authorization: Bearer dummy
Content-Type: application/json
Content-Length: 1175
{'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': "You are now the following python function: ```# This function takes a JSON string and ensures that it is parseable and fully compliant with the provided schema. If an object or field specified in the schema isn't contained within the correct JSON, it is omitted. The function also escapes any double quotes within JSON string values to ensure that they are valid. If the JSON string contains any None or NaN values, they are replaced with null before being parsed.\ndef fix_json(json_string: str, schema:str=None) -> str:```\n\nOnly respond with your `return` value."}, {'role': 'user', 'content': '\'\'\'\u200b</s>\'\'\', \'\'\'\n{\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n },\n "thoughts":\n {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\n- list that conveys\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n }\n}\n\'\'\''}], 'temperature': 0, 'max_tokens': None}
127.0.0.1 - - [20/May/2023 07:53:14] "POST /v1/chat/completions HTTP/1.1" 200 -
{'prompt': 'system: You are now the following python function: ```# This function takes a JSON string and ensures that it is parseable and fully compliant with the provided schema. If an object or field specified in the schema isn\'t contained within the correct JSON, it is omitted. The function also escapes any double quotes within JSON string values to ensure that they are valid. If the JSON string contains any None or NaN values, they are replaced with null before being parsed.\ndef fix_json(json_string: str, schema:str=None) -> str:```\n\nOnly respond with your `return` value.\n\nuser: \'\'\'\u200b</s>\'\'\', \'\'\'\n{\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n },\n "thoughts":\n {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\n- list that conveys\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n }\n}\n\'\'\'\nassistant: ', 'req_params': {'max_new_tokens': 2000, 'temperature': 0.001, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': False, 'echo': False, 'seed': -1, 'truncation_length': 65000, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1, 'early_stopping': False, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}, 'stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}
Output generated in 0.57 seconds (8.82 tokens/s, 5 tokens, context 275, seed 419260190)
{'response': '\u200b</s>'}Oobabooga Text Generation log (cleaned)
Host: 127.0.0.1:5001
User-Agent: OpenAI/v1 PythonBindings/0.27.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.2", "httplib": "requests", "lang": "python", "lang_version": "3.11.0", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
Authorization: Bearer dummy
Content-Type: application/json
Content-Length: 4076
{'model': 'gpt-3.5-turbo',
'messages':
[{
'role': 'system',
'content': 'You are DataFinderGPT, An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently. Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.
GOALS:
Constraints:
1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3. No user assistance
4. Exclusively use the commands listed below e.g. command_name
Commands:
1. analyze_code: Analyze Code, args: "code": "<full_code_string>"
2. execute_python_file: Execute Python File, args: "filename": "<filename>"
3. append_to_file: Append to file, args: "filename": "<filename>", "text": "<text>"
4. delete_file: Delete file, args: "filename": "<filename>"
5. list_files: List Files in Directory, args: "directory": "<directory>"
6. read_file: Read file, args: "filename": "<filename>"
7. write_to_file: Write to file, args: "filename": "<filename>", "text": "<text>"
8. google: Google Search, args: "query": "<query>"
9. improve_code: Get Improved Code, args: "suggestions": "<list_of_suggestions>", "code": "<full_code_string>"
10. send_tweet: Send Tweet, args: "tweet_text": "<tweet_text>"
11. browse_website: Browse Website, args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"
12. write_tests: Write Tests, args: "code": "<full_code_string>", "focus": "<list_of_focus_areas>"
13. delete_agent: Delete GPT Agent, args: "key": "<key>"
14. get_hyperlinks: Get text summary, args: "url": "<url>"
15. get_text_summary: Get text summary, args: "url": "<url>", "question": "<question>"
16. list_agents: List GPT Agents, args: () -> str
17. message_agent: Message GPT Agent, args: "key": "<key>", "message": "<message>"
18. start_agent: Start GPT Agent, args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"
19. task_complete: Task Complete (Shutdown), args: "reason": "<reason>"
20. make_api_call: "Make an API call", args: "host": "<host>", "endpoint": "<endpoint>", "method": "<method>", "query_params": "<query_params>", "body": "<body>", "headers": "<headers>"
21. Get number of astronauts: "get_num_astronauts", args:
Resources:
1. Internet access for searches and information gathering.
2. Long Term memory management.
3. GPT-3.5 powered Agents for delegation of simple tasks.
4. File output.
Performance Evaluation:
1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2. Constructively self-criticize your big-picture behavior constantly.
3. Reflect on past decisions and strategies to refine your approach.
4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.
5. Write all code to a file.
You should only respond in JSON format as described below
Response Format:
{
"thoughts": {
"text": "thought",
"reasoning": "reasoning",
"plan": "
- short bulleted
- list that conveys
- long-term plan",
"criticism": "constructive self-criticism",
"speak": "thoughts summary to say to user"
},
"command": {
"name": "command name",
"args": {
"arg name": "value"
}
}
}
Ensure the response can be parsed by Python json.loads'
},
{
'role': 'system',
'content': 'The current time and date is Sat May 20 07:53:08 2023'
},
{
'role': 'user',
'content': 'Determine which next command to use, and respond using the format specified above:'
}],
'temperature': 0.0,
'max_tokens': 2576
}
127.0.0.1 - - [20/May/2023 07:53:08] "POST /v1/chat/completions HTTP/1.1" 200 -
{
'prompt': 'system: You are DataFinderGPT, An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently. Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.
GOALS:
Constraints:
1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3. No user assistance
4. Exclusively use the commands listed below e.g. command_name
Commands:
1. analyze_code: Analyze Code, args: "code": "<full_code_string>"
2. execute_python_file: Execute Python File, args: "filename": "<filename>"
3. append_to_file: Append to file, args: "filename": "<filename>", "text": "<text>"
4. delete_file: Delete file, args: "filename": "<filename>"
5. list_files: List Files in Directory, args: "directory": "<directory>"
6. read_file: Read file, args: "filename": "<filename>"
7. write_to_file: Write to file, args: "filename": "<filename>", "text": "<text>"
8. google: Google Search, args: "query": "<query>"
9. improve_code: Get Improved Code, args: "suggestions": "<list_of_suggestions>", "code": "<full_code_string>"
10. send_tweet: Send Tweet, args: "tweet_text": "<tweet_text>"
11. browse_website: Browse Website, args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"
12. write_tests: Write Tests, args: "code": "<full_code_string>", "focus": "<list_of_focus_areas>"
13. delete_agent: Delete GPT Agent, args: "key": "<key>"
14. get_hyperlinks: Get text summary, args: "url": "<url>"
15. get_text_summary: Get text summary, args: "url": "<url>", "question": "<question>"
16. list_agents: List GPT Agents, args: () -> str
17. message_agent: Message GPT Agent, args: "key": "<key>", "message": "<message>"
18. start_agent: Start GPT Agent, args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"
19. task_complete: Task Complete (Shutdown), args: "reason": "<reason>"
20. make_api_call: "Make an API call", args: "host": "<host>", "endpoint": "<endpoint>", "method": "<method>", "query_params": "<query_params>", "body": "<body>", "headers": "<headers>"
21. Get number of astronauts: "get_num_astronauts", args:
Resources:
1. Internet access for searches and information gathering.
2. Long Term memory management.
3. GPT-3.5 powered Agents for delegation of simple tasks.
4. File output.
Performance Evaluation:
1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2. Constructively self-criticize your big-picture behavior constantly.
3. Reflect on past decisions and strategies to refine your approach.
4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.
5. Write all code to a file.
You should only respond in JSON format as described below
Response Format:
{
"thoughts": {
"text": "thought",
"reasoning": "reasoning",
"plan": "
- short bulleted
- list that conveys
- long-term plan",
"criticism": "constructive self-criticism",
"speak": "thoughts summary to say to user"
},
"command": {
"name": "command name",
"args": {
"arg name": "value"
}
}
}
Ensure the response can be parsed by Python json.loadsThe current time and date is Sat May 20 07:53:08 2023
user: Determine which next command to use, and respond using the format specified above:
assistant: ',
'req_params':
{
'max_new_tokens': 2576,
'temperature': 0.001,
'top_p': 1.0,
'top_k': 1,
'repetition_penalty': 1.18,
'encoder_repetition_penalty': 1.0,
'suffix': None,
'stream': False,
'echo': False,
'seed': -1,
'truncation_length': 65000,
'add_bos_token': True,
'do_sample': True,
'typical_p': 1.0,
'min_length': 0,
'no_repeat_ngram_size': 0,
'num_beams': 1,
'penalty_alpha': 0.0,
'length_penalty': 1,
'early_stopping': False,
'ban_eos_token': False,
'skip_special_tokens': True,
'custom_stopping_strings':
['
system:', '
user:', '
human:', '
assistant:', '###'
]
},
'stopping_strings':
['
system:', '
user:', '
human:', '
assistant:', '###'
]
}
Output generated in 3.94 seconds (1.27 tokens/s, 5 tokens, context 1111, seed 412636788)
{'response': '\u200b</s>'}
Host: 127.0.0.1:5001
User-Agent: OpenAI/v1 PythonBindings/0.27.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.2", "httplib": "requests", "lang": "python", "lang_version": "3.11.0", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
Authorization: Bearer dummy
Content-Type: application/json
Content-Length: 1175
{
'model': 'gpt-3.5-turbo',
'messages':
[
{
'role': 'system',
'content': "You are now the following python function: ```# This function takes a JSON string and ensures that it is parseable and fully compliant with the provided schema. If an object or field specified in the schema isn't contained within the correct JSON, it is omitted. The function also escapes any double quotes within JSON string values to ensure that they are valid. If the JSON string contains any None or NaN values, they are replaced with null before being parsed.
def fix_json(json_string: str, schema:str=None) -> str:```
Only respond with your `return` value."}, {'role': 'user', 'content': '\'\'\'\u200b</s>\'\'\', \'\'\'
{
"command":
{
"name": "command name",
"args":
{
"arg name": "value"
}
},
"thoughts":
{
"text": "thought",
"reasoning": "reasoning",
"plan": "
- short bulleted
- list that conveys
- long-term plan",
"criticism": "constructive self-criticism",
"speak": "thoughts summary to say to user"
}
}
\'\'\''
}
],
'temperature': 0,
'max_tokens': None
}
127.0.0.1 - - [20/May/2023 07:53:14] "POST /v1/chat/completions HTTP/1.1" 200 -
{
'prompt': 'system: You are now the following python function: ```# This function takes a JSON string and ensures that it is parseable and fully compliant with the provided schema. If an object or field specified in the schema isn\'t contained within the correct JSON, it is omitted. The function also escapes any double quotes within JSON string values to ensure that they are valid. If the JSON string contains any None or NaN values, they are replaced with null before being parsed.
def fix_json(json_string: str, schema:str=None) -> str:```
Only respond with your `return` value.
user: \'\'\'\u200b</s>\'\'\', \'\'\'
{
"command": {
"name": "command name",
"args": {
"arg name": "value"
}
},
"thoughts":
{
"text": "thought",
"reasoning": "reasoning",
"plan": "
- short bulleted
- list that conveys
- long-term plan",
"criticism": "constructive self-criticism",
"speak": "thoughts summary to say to user"
}
}
\'\'\'
assistant: ', 'req_params':
{
'max_new_tokens': 2000,
'temperature': 0.001,
'top_p': 1.0,
'top_k': 1,
'repetition_penalty': 1.18,
'encoder_repetition_penalty': 1.0,
'suffix': None,
'stream': False,
'echo': False,
'seed': -1,
'truncation_length': 65000,
'add_bos_token': True,
'do_sample': True,
'typical_p': 1.0,
'min_length': 0,
'no_repeat_ngram_size': 0,
'num_beams': 1,
'penalty_alpha': 0.0,
'length_penalty': 1,
'early_stopping': False,
'ban_eos_token': False,
'skip_special_tokens': True,
'custom_stopping_strings':
['
system:', '
user:', '
human:', '
assistant:', '###'
]
},
'stopping_strings':
['
system:', '
user:', '
human:', '
assistant:', '###'
]
}
Output generated in 0.57 seconds (8.82 tokens/s, 5 tokens, context 275, seed 419260190)
{'response': '\u200b</s>'}@Simplegram That was really helpful, thanks for the detailed logs. I think I see 2 problems I can fix (some space related problems) - after this change I get the following reply from wizard-mega:
{"thoughts":{"text":"I'm thinking about how I can help you with your project. What do you need assistance with?","reasoning":"I want to understand what you need help with so I can provide relevant guidance or resources."},"command":{"name":"help\_me","args":{"type":"project"}}}</s>The stray </s> on the wizard-mega model (AFAIK) is an artifact from a bug when making the model BTW, the fixed version is now called Manticore (try that instead).
Simplegram
commented
1 year ago @Simplegram That was really helpful, thanks for the detailed logs. I think I see 2 problems I can fix (some space related problems) - after this change I get the following reply from wizard-mega:
{"thoughts":{"text":"I'm thinking about how I can help you with your project. What do you need assistance with?","reasoning":"I want to understand what you need help with so I can provide relevant guidance or resources."},"command":{"name":"help\_me","args":{"type":"project"}}}</s>The stray
</s>on the wizard-mega model (AFAIK) is an artifact from a bug when making the model BTW, the fixed version is now called Manticore (try that instead).
Hello. I've tried TheBloke_Manticore-13B-GPTQ and here's the new log. Sorry I couldn't clean it at the moment.
Oobabooga Log
127.0.0.1 - - [22/May/2023 00:05:55] "POST /v1/chat/completions HTTP/1.1" 200 -
{'prompt': 'system: You are DataFinderGPT, An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently.\nYour decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.\n\nGOALS:\n\n\n\nConstraints:\n1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.\n2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.\n3. No user assistance\n4. Exclusively use the commands listed below e.g. command_name\n\nCommands:\n1. analyze_code: Analyze Code, args: "code": "<full_code_string>"\n2. execute_python_file: Execute Python File, args: "filename": "<filename>"\n3. append_to_file: Append to file, args: "filename": "<filename>", "text": "<text>"\n4. delete_file: Delete file, args: "filename": "<filename>"\n5. list_files: List Files in Directory, args: "directory": "<directory>"\n6. read_file: Read file, args: "filename": "<filename>"\n7. write_to_file: Write to file, args: "filename": "<filename>", "text": "<text>"\n8. google: Google Search, args: "query": "<query>"\n9. improve_code: Get Improved Code, args: "suggestions": "<list_of_suggestions>", "code": "<full_code_string>"\n10. send_tweet: Send Tweet, args: "tweet_text": "<tweet_text>"\n11. browse_website: Browse Website, args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"\n12. write_tests: Write Tests, args: "code": "<full_code_string>", "focus": "<list_of_focus_areas>"\n13. delete_agent: Delete GPT Agent, args: "key": "<key>"\n14. get_hyperlinks: Get text summary, args: "url": "<url>"\n15. get_text_summary: Get text summary, args: "url": "<url>", "question": "<question>"\n16. list_agents: List GPT Agents, args: () -> str\n17. message_agent: Message GPT Agent, args: "key": "<key>", "message": "<message>"\n18. start_agent: Start GPT Agent, args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"\n19. task_complete: Task Complete (Shutdown), args: "reason": "<reason>"\n20. make_api_call: "Make an API call", args: "host": "<host>", "endpoint": "<endpoint>", "method": "<method>", "query_params": "<query_params>", "body": "<body>", "headers": "<headers>"\n21. Get number of astronauts: "get_num_astronauts", args: \n\nResources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\n5. Write all code to a file.\n\nYou should only respond in JSON format as described below \nResponse Format: \n{\n "thoughts": {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\\n- list that conveys\\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n },\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n }\n} \nEnsure the response can be parsed by Python json.loads\nThe current time and date is Mon May 22 00:05:54 2023\nThis reminds you of these events from your past: \nI have been created without any significant developments occurring since my creation.\nCommand None returned: Error: \'NoneType\' object has no attribute \'lower\'\n\nuser: Determine which next command to use, and respond using the format specified above:\nassistant: The next command I would recommend is "execute_python_file".\nuser: Determine which next command to use, and respond using the format specified above:\nassistant:', 'req_params': {'max_new_tokens': 2507, 'temperature': 0.001, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': False, 'echo': False, 'seed': -1, 'truncation_length': 65000, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1, 'early_stopping': False, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}, 'stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}
Output generated in 0.87 seconds (17.31 tokens/s, 15 tokens, context 1196, seed 955019075)
{'response': 'The next command I would recommend is "append_to_file".'}
Host: 127.0.0.1:5001
User-Agent: OpenAI/v1 PythonBindings/0.27.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.2", "httplib": "requests", "lang": "python", "lang_version": "3.11.0", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
Authorization: Bearer dummy
Content-Type: application/json
Content-Length: 1222
{'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': "You are now the following python function: ```# This function takes a JSON string and ensures that it is parseable and fully compliant with the provided schema. If an object or field specified in the schema isn't contained within the correct JSON, it is omitted. The function also escapes any double quotes within JSON string values to ensure that they are valid. If the JSON string contains any None or NaN values, they are replaced with null before being parsed.\ndef fix_json(json_string: str, schema:str=None) -> str:```\n\nOnly respond with your `return` value."}, {'role': 'user', 'content': '\'\'\'The next command I would recommend is "append_to_file".\'\'\', \'\'\'\n{\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n },\n "thoughts":\n {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\n- list that conveys\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n }\n}\n\'\'\''}], 'temperature': 0, 'max_tokens': None}
127.0.0.1 - - [22/May/2023 00:05:56] "POST /v1/chat/completions HTTP/1.1" 200 -
{'prompt': 'system: You are now the following python function: ```# This function takes a JSON string and ensures that it is parseable and fully compliant with the provided schema. If an object or field specified in the schema isn\'t contained within the correct JSON, it is omitted. The function also escapes any double quotes within JSON string values to ensure that they are valid. If the JSON string contains any None or NaN values, they are replaced with null before being parsed.\ndef fix_json(json_string: str, schema:str=None) -> str:```\n\nOnly respond with your `return` value.\n\nuser: \'\'\'The next command I would recommend is "append_to_file".\'\'\', \'\'\'\n{\n "command": {\n "name": "command name",\n "args": {\n "arg name": "value"\n }\n },\n "thoughts":\n {\n "text": "thought",\n "reasoning": "reasoning",\n "plan": "- short bulleted\n- list that conveys\n- long-term plan",\n "criticism": "constructive self-criticism",\n "speak": "thoughts summary to say to user"\n }\n}\n\'\'\'\nassistant:', 'req_params': {'max_new_tokens': 2000, 'temperature': 0.001, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': False, 'echo': False, 'seed': -1, 'truncation_length': 65000, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1, 'early_stopping': False, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}, 'stopping_strings': ['\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}
Output generated in 3.72 seconds (25.56 tokens/s, 95 tokens, context 286, seed 1051207230)
{'response': "Your Python code looks good! Here's what you can do next:\n1. Add more commands to your script using the `add_command()` method. For example, add a command for adding new thoughts to the file.\n2. Use the `read_from_file()` method to read from the file instead of hardcoding the data into the script.\n3. Implement error handling by catching exceptions raised during execution of the script."}Auto-GPT Log
Plugins found: 2
--------------------
autogpt-api-tools: 0.1.0 - Allow AutoGPT to make API calls to outside services.
Auto-GPT-Space-Plugin: 0.1.3 - Auto-GPT Space Plugin: Get the number of astronauts in space in real-time..
Welcome back! Would you like me to return to being DataFinderGPT?
Asking user via keyboard...
Continue with the last settings?
Name: DataFinderGPT
Role: An intelligent search engine AI designed specifically for finding relevant data related to any given topic quickly and efficiently. It uses advanced natural language processing techniques to understand complex queries and provides accurate results within seconds.
Goals: []
API Budget: infinite
Continue (y/n): y
NOTE:All files/directories created by this agent can be found inside its workspace at: E:\Projects\cpy\llms\auto-gpt\Auto-GPT\autogpt\auto_gpt_workspace
DataFinderGPT has been created with the following details:
Name: DataFinderGPT
Role: An intelligent search engine AI designed specifically for finding relevant data related to any given topic quickly and efficiently. It uses advanced natural language processing techniques to understand complex queries and provides accurate results within seconds.
Goals:
Using memory of type: LocalCache
Using Browser: chrome
Error: The following AI output couldn't be converted to a JSON:
The next command I would recommend is "execute_python_file".
NEXT ACTION: COMMAND = None ARGUMENTS = None
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands, 'n' to exit program, or enter feedback for DataFinderGPT...
Asking user via keyboard...
Input:Well, good news is that the OpenAPI is working as intended. The bad news is that it's not doing what Auto-GPS asked, which is a known limit of the local models. I'm working on some improvements which may improve the obedience of the models, but so far it's challenging to get this kind of output (ie. "you are the following python function, output only json") from local models.
If AutoGPT switches to using "guidance" things may radically improve.
For now, this is not a bug in the openai extension though, just a limit of the model with Auto-GPT.
Simplegram
commented
1 year ago Well, good news is that the OpenAPI is working as intended. The bad news is that it's not doing what Auto-GPS asked, which is a known limit of the local models. I'm working on some improvements which may improve the obedience of the models, but so far it's challenging to get this kind of output (ie. "you are the following python function, output only json") from local models.
If AutoGPT switches to using "guidance" things may radically improve.
For now, this is not a bug in the openai extension though, just a limit of the model with Auto-GPT.
Hey, thanks a lot for your help. At least it's not a model bug anymore. I think I'm gonna close this.
matatonic
commented
1 year ago @Simplegram - Some additional improvements to the openai extension were just merged, I'd be curious if you have any better luck with Auto-GPT now, either with Manticore or maybe one of the llama-supercot models. I haven't had time to test any of those yet, so much is released every week!
cibernicola
commented
1 year ago I'm testing the same, but with langchain, having also some problems, I'll re-test everything again.
cibernicola
commented
1 year ago No way:
File "C:\Python310\lib\site-packages\langchain\agents\agent.py", line 444, in plan
return self.output_parser.parse(full_output)
File "C:\Python310\lib\site-packages\langchain\agents\mrkl\output_parser.py", line 27, in parse
raise OutputParserException(
langchain.schema.OutputParserException: Could not parse LLM output: `Thought: What is the head of df?`from webUI:
\nQuestion: jkljlk\n", 'req_params': {'max_new_tokens': 256, 'temperature': 0.001, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': False, 'echo': False, 'seed': -1, 'truncation_length': 2048, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'epsilon_cutoff': 0, 'eta_cutoff': 0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1, 'early_stopping': False, 'mirostat_mode': 0, 'mirostat_tau': 5, 'mirostat_eta': 0.1, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\nObservation:', '\n\tObservation:', '\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}, 'stopping_strings': ['\nObservation:', '\n\tObservation:', '\nsystem:', '\nuser:', '\nhuman:', '\nassistant:', '\n###']}
Output generated in 2.33 seconds (4.72 tokens/s, 11 tokens, context 1646, seed 1585307932)
{'response': 'Thought: What is the head of df?'}@Simplegram - Some additional improvements to the openai extension were just merged, I'd be curious if you have any better luck with Auto-GPT now, either with Manticore or maybe one of the llama-supercot models. I haven't had time to test any of those yet, so much is released every week!
Sadly still no luck after updating :(
Models I tried:
I'll keep experimenting.
Describe the bug
Auto-GPT can't convert any output from the API to JSON
Settings I've tried:
Models I've tried:
Is there an existing issue for this?
Reproduction
Name: DataFinderGPT Role: An intelligent search engine AI designed specifically for finding relevant and accurate data related to any given topic quickly and efficiently. Goals: [] API Budget: infinite
Screenshot
Logs
System Info
EDIT: The problem with OpenedAI API has been resolved using Manticore-13B. Now the issue is up to Auto-GPT to resolve.