search

Significant-Gravitas / AutoGPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
https://agpt.co
MIT License
167.22k stars 44.19k forks source link

Add Proxy Support in AutoGPT #6944

Open astraiosmem opened 7 months ago

astraiosmem commented 7 months ago

⚠️ Search for existing issues first ⚠️

Which Operating System are you using?

Linux

Which version of AutoGPT are you using?

Latest Release

Do you use OpenAI GPT-3 or GPT-4?

GPT-3.5

Which area covers your issue best?

Installation and setup

Describe your issue.

Excuse me, does anyone know how to solve this?

OS:Debian 13

description

I have an Xray proxy configured with a SOCKS port of 10808, since I'm located in mainland China, and I must use a proxy to bypass the Great Firewall (GFW), so turning off proxy will not work for me, I noticed that the url.scheme value passed by the system is 'socks', while the detection code expects 'socks5'. the code related is as follows (in /home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_config.py):

class Proxy:
    def __init__(
        self,
        url: URLTypes,
        *,
        auth: typing.Optional[typing.Tuple[str, str]] = None,
        headers: typing.Optional[HeaderTypes] = None,
    ):
        url = URL(url)
        headers = Headers(headers)

        if url.scheme not in ("http", "https", "socks5"):
            raise ValueError(f"Unknown scheme for proxy URL {url!r}")

        if url.username or url.password:
            # Remove any auth credentials from the URL.
            auth = (url.username, url.password)
            url = url.copy_with(username=None, password=None)

        self.url = url
        self.auth = auth
        self.headers = headers

This results in a ValueError: Unknown scheme for proxy URL URL('socks://127.0.0.1:10808/')

the complete info:

astraios@debian ~/A/a/autogpt (master)> ./autogpt.sh run --gpt3only                                                                                                                                         (base) 
Missing packages:
auto-gpt-plugin-template (*) @ git+https://github.com/Significant-Gravitas/Auto-GPT-Plugin-Template@0.1.0, autogpt-forge (*) @ git+https://github.com/Significant-Gravitas/AutoGPT.git@ab05b7ae70754c063909#subdirectory=autogpts/forge, boto3 (>=1.33.6,<2.0.0), charset-normalizer (>=3.1.0,<4.0.0), colorama (>=0.4.6,<0.5.0), docker, en-core-web-sm (*) @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0-py3-none-any.whl, fastapi (>=0.109.1,<0.110.0), ftfy (>=6.1.1,<7.0.0), gitpython (>=3.1.32,<4.0.0), google-api-python-client, gTTS (>=2.3.1,<3.0.0), hypercorn (>=0.14.4,<0.15.0), inflection, jsonschema, numpy, openai (>=1.7.2,<2.0.0), orjson (>=3.8.10,<4.0.0), pinecone-client (>=2.2.1,<3.0.0), playsound (>=1.2.2,<1.3.0), prompt_toolkit (>=3.0.38,<4.0.0), pylatexenc, pypdf (>=3.1.0,<4.0.0), python-docx, python-dotenv (>=1.0.0,<2.0.0), pyyaml (>=6.0,<7.0), readability-lxml (>=0.8.1,<0.9.0), redis, sentry-sdk (>=1.40.4,<2.0.0), spacy (>=3.0.0,<4.0.0), tenacity (>=8.2.2,<9.0.0), tiktoken (>=0.5.0,<0.6.0), webdriver-manager, openapi-python-client (>=0.14.0,<0.15.0), google-cloud-logging (>=3.8.0,<4.0.0), google-cloud-storage (>=2.13.0,<3.0.0), psycopg2-binary (>=2.9.9,<3.0.0)

Installing dependencies from lock file

No dependencies to install or update

Installing the current project: agpt (0.5.0)

Finished installing packages! Starting AutoGPT...

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/AutoGPT/autogpts/autogpt/autogpt/app/cli.py", line 179, in run
    from autogpt.app.main import run_auto_gpt
  File "/home/astraios/AutoGPT/autogpts/autogpt/autogpt/app/main.py", line 16, in <module>
    from forge.sdk.db import AgentDB
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/forge/sdk/__init__.py", line 5, in <module>
    from ..llm import chat_completion_request, create_embedding_request, transcribe_audio
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/forge/llm.py", line 3, in <module>
    from litellm import AuthenticationError, InvalidRequestError, ModelResponse, acompletion
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/litellm/__init__.py", line 536, in <module>
    from .main import *  # type: ignore
    ^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/litellm/main.py", line 94, in <module>
    openai_text_completions = OpenAITextCompletion()
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/litellm/llms/openai.py", line 795, in __init__
    self._client_session = self.create_client_session()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/litellm/llms/base.py", line 14, in create_client_session
    _client_session = httpx.Client()
                      ^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 670, in __init__
    proxy_map = self._get_proxy_map(proxies, allow_env_proxies)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 216, in _get_proxy_map
    return {
           ^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 217, in <dictcomp>
    key: None if url is None else Proxy(url=url)
                                  ^^^^^^^^^^^^^^
  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_config.py", line 336, in __init__
    raise ValueError(f"Unknown scheme for proxy URL {url!r}")
ValueError: Unknown scheme for proxy URL URL('socks://127.0.0.1:10808/')

after comment the two line and rerun ./autogpt.sh run --gpt3only, the error changed:

       # if url.scheme not in ("http", "https", "socks5"):
        #     raise ValueError(f"Unknown scheme for proxy URL {url!r}")

error:

  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 182, in __init__
    raise ValueError(
ValueError: Proxy protocol must be either 'http', 'https', or 'socks5', but got 'socks'.

the related code in "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py":

        elif proxy.url.scheme == "socks5":
            try:
                import socksio  # noqa
            except ImportError:  # pragma: no cover
                raise ImportError(
                    "Using SOCKS proxy, but the 'socksio' package is not installed. "
                    "Make sure to install httpx using `pip install httpx[socks]`."
                ) from None

and I tried change it to:

        elif proxy.url.scheme in ("socks5","socks"):
            try:
                import socksio  # noqa
            except ImportError:  # pragma: no cover
                raise ImportError(
                    "Using SOCKS proxy, but the 'socksio' package is not installed. "
                    "Make sure to install httpx using `pip install httpx[socks]`."
                ) from None

and then the error:

  File "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 161, in __init__
    raise ImportError(
ImportError: Using SOCKS proxy, but the 'socksio' package is not installed. Make sure to install httpx using `pip install httpx[socks]`

I execute the command successfully:

astraios@debian ~/A/a/autogpt (master)> pip install httpx[socks]                                                                                                                                            (base) 
Requirement already satisfied: httpx[socks] in /home/astraios/miniconda3/lib/python3.11/site-packages (0.27.0)
Requirement already satisfied: anyio in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpx[socks]) (4.3.0)
Requirement already satisfied: certifi in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpx[socks]) (2023.11.17)
Requirement already satisfied: httpcore==1.* in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpx[socks]) (1.0.4)
Requirement already satisfied: idna in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpx[socks]) (3.4)
Requirement already satisfied: sniffio in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpx[socks]) (1.3.1)
Requirement already satisfied: socksio==1.* in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpx[socks]) (1.0.0)
Requirement already satisfied: h11<0.15,>=0.13 in /home/astraios/miniconda3/lib/python3.11/site-packages (from httpcore==1.*->httpx[socks]) (0.14.0)

but the error still exist, and I also tried pip install socksio

Any help will be greatly appreciated

Upload Activity Log Content

No response

Upload Error Log Content

No response

astraiosmem commented 7 months ago

after adding the following code to the "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py " file, the problem is solved:

import sys
sys.path.append("/home/astraios/miniconda3/lib/python3.11/site-packages")

the address can be located by pip show socksio

MooMooHorse commented 6 months ago

I simply changed https_proxy to socks5, and it also solves the problem. Thanks for looking into _config.py file for us.

$env:https_proxy = "socks5://127.0.0.1:10808"

instead of

$env:https_proxy = "socks://127.0.0.1:10808"

github-actions[bot] commented 4 months ago

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

letswish commented 4 months ago

修改创建所创建的虚拟环境的python中的httpx包的内容。按照题主的路径是：home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_config.py。查找到Proxy类下的url = URL(url)行。在此行前添加url = "socks5://127.0.0.1:10808"（10808为题主端口号)保存即可。

ntindle commented 3 months ago

We would be interested in having someone help out with adding this

github-actions[bot] commented 2 months ago

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

tedy-li commented 1 month ago

$env:https_proxy = "socks5://127.0.0.1:10808"

where to add this code?

tedy-li commented 1 month ago

after adding the following code to the "/home/astraios/.cache/pypoetry/virtualenvs/agpt-47GbBSF9-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py " file, the problem is solved:

import sys
sys.path.append("/home/astraios/miniconda3/lib/python3.11/site-packages")

the address can be located by pip show socksio

i meet the same questions could you send all changes points to sovles this problem. thanks