search

geekan / MetaGPT

🌟 The Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming
https://deepwisdom.ai/
MIT License
43.5k stars 5.18k forks source link

using local models instead of openAI? #180

Open dasam opened 1 year ago

dasam commented 1 year ago

I am using GPT4All to pull various different LLM images, it can start a local server for API access ( https://docs.gpt4all.io/gpt4all_chat.html ) would it possible to customize to use that API instead , to go truly open source?

voidking commented 1 year ago

Yes, metagpt can do that. But your should modify some code to suit your LLM API.

bartanderson commented 1 year ago

I would be interested to know what code. I started by making the config use my mini_orca configuration. I started running and ran into this issue and am wondering how much is OpenAI specific and where to change what is. Maybe I am missing some config, but if I am needing code to write, a summary of the files needed to be replaced or rewritten with how to access that path would be great. Here is where I ran into an authentication after initially bypassing it by having my own model. (MetaGPT) D:\Github\MetaGPT>python startup.py "Write a cli snake game based on pygame" Found model file at d:\\models\\GPT4ALL\orca-mini-3b.ggmlv3.q4_0.bin llama.cpp: loading model from d:\\models\\GPT4ALL\orca-mini-3b.ggmlv3.q4_0.bin llama_model_load_internal: format = ggjt v3 (latest) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 2048 llama_model_load_internal: n_embd = 3200 llama_model_load_internal: n_mult = 240 llama_model_load_internal: n_head = 32 llama_model_load_internal: n_layer = 26 llama_model_load_internal: n_rot = 100 llama_model_load_internal: ftype = 2 (mostly Q4_0) llama_model_load_internal: n_ff = 8640 llama_model_load_internal: n_parts = 1 llama_model_load_internal: model size = 3B llama_model_load_internal: ggml ctx size = 0.06 MB llama_model_load_internal: mem required = 2862.72 MB (+ 682.00 MB per state) llama_new_context_with_model: kv self size = 650.00 MB 2023-08-10 13:55:31.896 | INFO | metagpt.config:init:46 - Config loading done. 2023-08-10 13:55:35.595 | INFO | metagpt.software_company:invest:39 - Investment: $3.0. 2023-08-10 13:55:35.599 | INFO | metagpt.roles.role:_act:167 - Alice(Product Manager): ready to WritePRD Traceback (most recent call last): File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity_asyncio.py", line 50, in call result = await fn(*args, kwargs) File "D:\Github\MetaGPT\metagpt\actions\action.py", line 57, in _aask_v1 content = await self.llm.aask(prompt, system_msgs) File "D:\Github\MetaGPT\metagpt\provider\base_gpt_api.py", line 44, in aask rsp = await self.acompletion_text(message, stream=True) File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity_asyncio.py", line 88, in async_wrapped return await fn(*args, *kwargs) File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity_asyncio.py", line 47, in call do = self.iter(retry_state=retry_state) File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity__init.py", line 314, in iter return fut.result() File "C:\Users\bartl.pyenv\pyenv-win\versions\3.10.11\lib\concurrent\futures_base.py", line 451, in result return self.get_result() File "C:\Users\bartl.pyenv\pyenv-win\versions\3.10.11\lib\concurrent\futures_base.py", line 403, in get_result raise self._exception File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity_asyncio.py", line 50, in call__ result = await fn(args, kwargs) File "D:\Github\MetaGPT\metagpt\provider\openai_api.py", line 222, in acompletion_text return await self._achat_completion_stream(messages) File "D:\Github\MetaGPT\metagpt\provider\openai_api.py", line 151, in _achat_completion_stream response = await openai.ChatCompletion.acreate(*self._cons_kwargs(messages), stream=True) File "D:\Github\MetaGPT\lib\site-packages\openai-0.27.8-py3.10.egg\openai\api_resources\chat_completion.py", line 45, in acreate return await super().acreate(args, **kwargs) File "D:\Github\MetaGPT\lib\site-packages\openai-0.27.8-py3.10.egg\openai\api_resources\abstract\engine_apiresource.py", line 217, in acreate response, , api_key = await requestor.arequest( File "D:\Github\MetaGPT\lib\site-packages\openai-0.27.8-py3.10.egg\openai\api_requestor.py", line 382, in arequest resp, got_stream = await self._interpret_async_response(result, stream) File "D:\Github\MetaGPT\lib\site-packages\openai-0.27.8-py3.10.egg\openai\api_requestor.py", line 726, in _interpret_async_response self._interpret_response_line( File "D:\Github\MetaGPT\lib\site-packages\openai-0.27.8-py3.10.egg\openai\api_requestor.py", line 763, in _interpret_response_line raise self.handle_error_response( openai.error.AuthenticationError: Incorrect API key provided: dummyval. You can find your API key at https://platform.openai.com/account/api-keys.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\Github\MetaGPT\startup.py", line 40, in fire.Fire(main) File "D:\Github\MetaGPT\lib\site-packages\fire-0.4.0-py3.10.egg\fire\core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "D:\Github\MetaGPT\lib\site-packages\fire-0.4.0-py3.10.egg\fire\core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "D:\Github\MetaGPT\lib\site-packages\fire-0.4.0-py3.10.egg\fire\core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, kwargs) File "D:\Github\MetaGPT\startup.py", line 36, in main asyncio.run(startup(idea, investment, n_round, code_review, run_tests)) File "C:\Users\bartl.pyenv\pyenv-win\versions\3.10.11\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Users\bartl.pyenv\pyenv-win\versions\3.10.11\lib\asyncio\base_events.py", line 649, in run_until_complete return future.result() File "D:\Github\MetaGPT\startup.py", line 24, in startup await company.run(n_round=n_round) File "D:\Github\MetaGPT\metagpt\software_company.py", line 60, in run await self.environment.run() File "D:\Github\MetaGPT\metagpt\environment.py", line 67, in run await asyncio.gather(futures) File "D:\Github\MetaGPT\metagpt\roles\role.py", line 240, in run rsp = await self._react() File "D:\Github\MetaGPT\metagpt\roles\role.py", line 209, in _react return await self._act() File "D:\Github\MetaGPT\metagpt\roles\role.py", line 168, in _act response = await self._rc.todo.run(self._rc.important_memory) File "D:\Github\MetaGPT\metagpt\actions\write_prd.py", line 145, in run prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING) File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity_asyncio.py", line 88, in async_wrapped return await fn(args, kwargs) File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity_asyncio.py", line 47, in call do = self.iter(retry_state=retry_state) File "D:\Github\MetaGPT\lib\site-packages\tenacity-8.2.2-py3.10.egg\tenacity__init__.py", line 326, in iter raise retry_exc from fut.exception() tenacity.RetryError: RetryError[<Future at 0x2168a720f70 state=finished raised AuthenticationError>] ------------------------------(disclaimer, I am not much of a programmer but Claude and ChatGPT enable me as a meat puppet to to many things)

voidking commented 1 year ago

@geekan @stellaHSR

bartanderson commented 1 year ago

I started trying to use ChatGPT to use the base class and the configuration to add something in but I think it is beyond our ability. So I will have to wait for someone who is smarter to do it or I will need to be a bit smarter about the architecture to do it. I guess that's the same thing, really :)

stellaHSR commented 1 year ago

openai.error.AuthenticationError: Incorrect API key provided: dummyval. You can find your API key at https://platform.openai.com/account/api-keys.

check your key

brian-toner commented 1 year ago

@bartanderson I am attempting the same thing. You'll need to change the server address for the api to your local host:

OPENAI_API_BASE: "http://127.0.0.1:8000/v1" #Use the port that your local gpt server gives you
OPENAI_API_KEY: "xxxxxxxxxxxxxxxxxxxxxxxxxx" #this is just a dummy value

This can be found in: MetaGPT/config/config.yaml

brian-toner commented 1 year ago

However, when I run it locally, I get the following error:

> python startup.py "Write a cli snake game based on pygame" --code_review True
2023-08-14 16:24:22.781 | INFO     | metagpt.config:__init__:44 - Config loading done.
2023-08-14 16:24:23.819 | INFO     | metagpt.software_company:invest:39 - Investment: $3.0.
2023-08-14 16:24:23.819 | INFO     | metagpt.roles.role:_act:167 - Alice(Product Manager): ready to WritePRD
You are a Product Manager, named Alice, your goal is Efficiently create a successful product, and the constraint is . 
You are a Product Manager, named Alice, your goal is Efficiently create a successful product, and the constraint is . 
Traceback (most recent call last):
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/metagpt/actions/action.py", line 60, in _aask_v1
    parsed_data = OutputParser.parse_data_with_mapping(content, output_data_mapping)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/metagpt/utils/common.py", line 121, in parse_data_with_mapping
    block_dict = cls.parse_blocks(data)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/metagpt/utils/common.py", line 43, in parse_blocks
    block_title, block_content = block.split("\n", 1)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<redacted>/MetaGPT/startup.py", line 40, in <module>
    fire.Fire(main)
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/startup.py", line 36, in main
    asyncio.run(startup(idea, investment, n_round, code_review, run_tests))
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/startup.py", line 24, in startup
    await company.run(n_round=n_round)
  File "<redacted>/MetaGPT/metagpt/software_company.py", line 60, in run
    await self.environment.run()
  File "<redacted>/MetaGPT/metagpt/environment.py", line 67, in run
    await asyncio.gather(*futures)
  File "<redacted>/MetaGPT/metagpt/roles/role.py", line 240, in run
    rsp = await self._react()
          ^^^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/metagpt/roles/role.py", line 209, in _react
    return await self._act()
           ^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/metagpt/roles/role.py", line 168, in _act
    response = await self._rc.todo.run(self._rc.important_memory)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/MetaGPT/metagpt/actions/write_prd.py", line 145, in run
    prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7f11aebbdc50 state=finished raised ValueError>]
bartanderson commented 1 year ago

I don't intend to use openai, does this have its own local ai that I missed?

On Mon, Aug 14, 2023, 3:31 PM Brian Toner @.***> wrote:

However, when I run it locally, I get the following error:

python startup.py "Write a cli snake game based on pygame" --code_review True 2023-08-14 16:24:22.781 | INFO | metagpt.config:init:44 - Config loading done. 2023-08-14 16:24:23.819 | INFO | metagpt.software_company:invest:39 - Investment: $3.0. 2023-08-14 16:24:23.819 | INFO | metagpt.roles.role:_act:167 - Alice(Product Manager): ready to WritePRD You are a Product Manager, named Alice, your goal is Efficiently create a successful product, and the constraint is . You are a Product Manager, named Alice, your goal is Efficiently create a successful product, and the constraint is . Traceback (most recent call last): File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in call result = await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/MetaGPT/metagpt/actions/action.py", line 60, in _aask_v1 parsed_data = OutputParser.parse_data_with_mapping(content, output_data_mapping) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/MetaGPT/metagpt/utils/common.py", line 121, in parse_data_with_mapping block_dict = cls.parse_blocks(data) ^^^^^^^^^^^^^^^^^^^^^^ File "/MetaGPT/metagpt/utils/common.py", line 43, in parse_blocks block_title, block_content = block.split("\n", 1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: not enough values to unpack (expected 2, got 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/MetaGPT/startup.py", line 40, in fire.Fire(main) File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/MetaGPT/startup.py", line 36, in main asyncio.run(startup(idea, investment, n_round, code_review, run_tests)) File "/miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/MetaGPT/startup.py", line 24, in startup await company.run(n_round=n_round) File "/MetaGPT/metagpt/software_company.py", line 60, in run await self.environment.run() File "/MetaGPT/metagpt/environment.py", line 67, in run await asyncio.gather(futures) File "/MetaGPT/metagpt/roles/role.py", line 240, in run rsp = await self._react() ^^^^^^^^^^^^^^^^^^^ File "/MetaGPT/metagpt/roles/role.py", line 209, in _react return await self._act() ^^^^^^^^^^^^^^^^^ File "/MetaGPT/metagpt/roles/role.py", line 168, in _act response = await self._rc.todo.run(self._rc.important_memory) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/MetaGPT/metagpt/actions/write_prd.py", line 145, in run prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped return await fn(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in call do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/init.py", line 326, in iter raise retry_exc from fut.exception() tenacity.RetryError: RetryError[<Future at 0x7f11aebbdc50 state=finished raised ValueError>]

— Reply to this email directly, view it on GitHub https://github.com/geekan/MetaGPT/issues/180#issuecomment-1678017048, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMXAKGKXC6KR7QJWZOT3FTXVKDK3ANCNFSM6AAAAAA3KIYTSI . You are receiving this because you were mentioned.Message ID: @.***>

brian-toner commented 1 year ago

@bartanderson You'll need to run your own local, OpenAI-like server running the llm of your choice. For instance, I am testing on llama-cpp-python's sever mode. In my case, my server is running on port 8000, on my localhost.

bartanderson commented 1 year ago

Yup, I have my eye on localai, which has a build problem currently I am trying to get help with and simpleai which may be too simple yet but who knows. So many projects to learn about and try. Did you do much other than expose it? The instructions I saw didn't say anything about how to interface to other than openai. I'd be interested in what you did.

On Tue, Aug 15, 2023 at 10:45 AM Brian Toner @.***> wrote:

@bartanderson https://github.com/bartanderson You'll need to run your own local, OpenAI-like server running the llm of your choice. For instance, I am testing on llama-cpp-python's sever mode. In my case, my server is running on port 8000, on my localhost.

— Reply to this email directly, view it on GitHub https://github.com/geekan/MetaGPT/issues/180#issuecomment-1679168119, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMXAKEVUASJYZ4IMIARLDTXVOKQLANCNFSM6AAAAAA3KIYTSI . You are receiving this because you were mentioned.Message ID: @.***>

Amit-Khetan-70 commented 1 year ago

Hello, @dasam joining weeks late in the wagon, could you please help me understand if any locally running models are providing desired results?

As I see, @bartanderson & @brian-toner have been discussing on the same note. Bests.

brian-toner commented 1 year ago

@bartanderson I can hit my OpenAI-like server fine. I've been debugging MetaGPT to try and sort out where my issue is coming from. But unfortunately I don't have a lot of time. I'll probably have more time over the weekend to see if I can figure it out.

brian-toner commented 1 year ago

Here is an update on my progress. I can validate that MetaGPT does work on a local server. It seems the "stop" field in the request body can take on a null value. The OpenAI like api I am using (llama-cpp) doesn't handle nulls properly in that field. Upon fixing it on the server side, Metagpt seems to work. Though I did encounter another error:

2023-08-18 20:57:35.198 | INFO     | metagpt.provider.openai_api:update_cost:81 - Total running cost: $0.073 | Max budget: $3.000 | Current cost: $0.047, prompt_tokens: 845, completion_tokens: 361
Traceback (most recent call last):
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 53, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/actions/action.py", line 62, in _aask_v1
    instruct_content = output_class(**parsed_data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 8 validation errors for prd
Original Requirements
  field required (type=value_error.missing)
Product Goals
  field required (type=value_error.missing)
User Stories
  field required (type=value_error.missing)
Competitive Quadrant Chart
  field required (type=value_error.missing)
Requirement Analysis
  field required (type=value_error.missing)
Requirement Pool
  field required (type=value_error.missing)
UI Design draft
  field required (type=value_error.missing)
Anything UNCLEAR
  field required (type=value_error.missing)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/<redacted>/MetaGPT/startup.py", line 40, in <module>
    fire.Fire(main)
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/startup.py", line 36, in main
    asyncio.run(startup(idea, investment, n_round, code_review, run_tests))
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/startup.py", line 24, in startup
    await company.run(n_round=n_round)
  File "/<redacted>/MetaGPT/metagpt/software_company.py", line 60, in run
    await self.environment.run()
  File "/<redacted>/MetaGPT/metagpt/environment.py", line 67, in run
    await asyncio.gather(*futures)
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 240, in run
    rsp = await self._react()
          ^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 209, in _react
    return await self._act()
           ^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 168, in _act
    response = await self._rc.todo.run(self._rc.important_memory)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/actions/write_prd.py", line 145, in run
    prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 93, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7fbf25361510 state=finished raised ValidationError>]

This can be worked around by increasing the number of stop attempts in action.py (around line 49) changing

    @retry(stop=stop_after_attempt(2), wait=wait_fixed(1))

to something like:

    @retry(stop=stop_after_attempt(99), wait=wait_fixed(1))
0xkr4t0s commented 1 year ago

Have you tried using llama2-wrapper It’d let you use an OpenAI compatible API for your models.

TinaTiel commented 12 months ago

Here is an update on my progress. I can validate that MetaGPT does work on a local server. It seems the "stop" field in the request body can take on a null value. The OpenAI like api I am using (llama-cpp) doesn't handle nulls properly in that field. Upon fixing it on the server side, Metagpt seems to work. Though I did encounter another error:

2023-08-18 20:57:35.198 | INFO     | metagpt.provider.openai_api:update_cost:81 - Total running cost: $0.073 | Max budget: $3.000 | Current cost: $0.047, prompt_tokens: 845, completion_tokens: 361
Traceback (most recent call last):
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 53, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/actions/action.py", line 62, in _aask_v1
    instruct_content = output_class(**parsed_data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 8 validation errors for prd
Original Requirements
  field required (type=value_error.missing)
Product Goals
  field required (type=value_error.missing)
User Stories
  field required (type=value_error.missing)
Competitive Quadrant Chart
  field required (type=value_error.missing)
Requirement Analysis
  field required (type=value_error.missing)
Requirement Pool
  field required (type=value_error.missing)
UI Design draft
  field required (type=value_error.missing)
Anything UNCLEAR
  field required (type=value_error.missing)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/<redacted>/MetaGPT/startup.py", line 40, in <module>
    fire.Fire(main)
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/startup.py", line 36, in main
    asyncio.run(startup(idea, investment, n_round, code_review, run_tests))
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/startup.py", line 24, in startup
    await company.run(n_round=n_round)
  File "/<redacted>/MetaGPT/metagpt/software_company.py", line 60, in run
    await self.environment.run()
  File "/<redacted>/MetaGPT/metagpt/environment.py", line 67, in run
    await asyncio.gather(*futures)
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 240, in run
    rsp = await self._react()
          ^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 209, in _react
    return await self._act()
           ^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 168, in _act
    response = await self._rc.todo.run(self._rc.important_memory)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/actions/write_prd.py", line 145, in run
    prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 93, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7fbf25361510 state=finished raised ValidationError>]

This can be worked around by increasing the number of stop attempts in action.py (around line 49) changing

    @retry(stop=stop_after_attempt(2), wait=wait_fixed(1))

to something like:

    @retry(stop=stop_after_attempt(99), wait=wait_fixed(1))

Do you have that fix in llamacpp python server? Or could you make a PR there?

Would run llama2-wrapper (and it does work) but it doesn't support cuBLAS so it ends up running the model on the CPU instead of the GPU

brian-toner commented 12 months ago

@TinaTiel unfortunately the code I used wasn't merged to main. The code I worked from was: https://raw.githubusercontent.com/ggerganov/llama.cpp/d8a8d0e536cfdaca0135f22d43fda80dc5e47cd8/examples/server/api_like_OAI.py

I found this linked on stack exchange, I think, explaining that there was a bug in the open-ai-like api, so I didn't work off of the main branch. I was hoping that it would get merged and then I would push up my changes, but it doesn't seem like that is going to happen.

From this obscure branch, I changed the line:

  if(is_present(body, "stop")): postData["stop"] += body["stop"]

to

  if(is_present(body, "stop") and body["stop"] is not None ): postData["stop"] += body["stop"]
Aniruddha120 commented 11 months ago

@bartanderson You'll need to run your own local, OpenAI-like server running the llm of your choice. For instance, I am testing on llama-cpp-python's sever mode. In my case, my server is running on port 8000, on my localhost.

Some questions:

  1. Are still using the LLAMA on meta GPT without any problem?

  2. What did you just do? only change the link and API key?

  3. is there any option to access LLAMA from remote API?

  4. Are you using the actual LLAMA or fine-tuned code LLAMA?

itlackey commented 11 months ago

I have local code llama running via FastChat OpenAI API and would love to use MetaGPT with it. Are there any other changes other than config and stop attempts that need to be made?

itlackey commented 11 months ago

Here is an update on my progress. I can validate that MetaGPT does work on a local server. It seems the "stop" field in the request body can take on a null value. The OpenAI like api I am using (llama-cpp) doesn't handle nulls properly in that field. Upon fixing it on the server side, Metagpt seems to work. Though I did encounter another error:

2023-08-18 20:57:35.198 | INFO     | metagpt.provider.openai_api:update_cost:81 - Total running cost: $0.073 | Max budget: $3.000 | Current cost: $0.047, prompt_tokens: 845, completion_tokens: 361
Traceback (most recent call last):
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 53, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/actions/action.py", line 62, in _aask_v1
    instruct_content = output_class(**parsed_data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 8 validation errors for prd
Original Requirements
  field required (type=value_error.missing)
Product Goals
  field required (type=value_error.missing)
User Stories
  field required (type=value_error.missing)
Competitive Quadrant Chart
  field required (type=value_error.missing)
Requirement Analysis
  field required (type=value_error.missing)
Requirement Pool
  field required (type=value_error.missing)
UI Design draft
  field required (type=value_error.missing)
Anything UNCLEAR
  field required (type=value_error.missing)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/<redacted>/MetaGPT/startup.py", line 40, in <module>
    fire.Fire(main)
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/startup.py", line 36, in main
    asyncio.run(startup(idea, investment, n_round, code_review, run_tests))
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/startup.py", line 24, in startup
    await company.run(n_round=n_round)
  File "/<redacted>/MetaGPT/metagpt/software_company.py", line 60, in run
    await self.environment.run()
  File "/<redacted>/MetaGPT/metagpt/environment.py", line 67, in run
    await asyncio.gather(*futures)
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 240, in run
    rsp = await self._react()
          ^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 209, in _react
    return await self._act()
           ^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/roles/role.py", line 168, in _act
    response = await self._rc.todo.run(self._rc.important_memory)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>/MetaGPT/metagpt/actions/write_prd.py", line 145, in run
    prd = await self._aask_v1(prompt, "prd", OUTPUT_MAPPING)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 93, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<redacted>miniconda3/envs/metagpt/lib/python3.11/site-packages/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7fbf25361510 state=finished raised ValidationError>]

This can be worked around by increasing the number of stop attempts in action.py (around line 49) changing

    @retry(stop=stop_after_attempt(2), wait=wait_fixed(1))

to something like:

    @retry(stop=stop_after_attempt(99), wait=wait_fixed(1))

Do you have that fix in llamacpp python server? Or could you make a PR there?

Would run llama2-wrapper (and it does work) but it doesn't support cuBLAS so it ends up running the model on the CPU instead of the GPU

I have a docker image that uses the Intel CLBLAST llama.cpp to host an open ai model. You may be able to mod it to build llama.cpp with cuBlas.

https://github.com/itlackey/ipex-arc-fastchat