search

danielgross / localpilot

MIT License
3.32k stars 141 forks source link

running a model other than CoPilot crashes #5

Open deven367 opened 9 months ago

deven367 commented 9 months ago 
(fun) ➜  localpilot git:(main) ✗ python app.py
Running server...
DEBUG:asyncio:Using selector: KqueueSelector
INFO:     Started server process [8436]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
DEBUG:root:Running: python3 -m llama_cpp.server --model /Users/deven367/models/codellama-7b.Q5_K_S.gguf --n_gpu_layers 1 --n_ctx 4096
INFO:     127.0.0.1:49968 - "POST /set_target HTTP/1.1" 200 OK
Successfully sent selection: CodeLlama-7b.
DEBUG:root:Current state: {'url': 'https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q5_K_S.gguf', 'type': 'local', 'filename': 'codellama-7b.Q5_K_S.gguf'}
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/certifi/cacert.pem'
DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=8000 local_address=None timeout=30 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError('All connection attempts failed'))
INFO:     127.0.0.1:49970 - "POST /v1/engines/copilot-codex/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
  | ExceptionGroup: multiple connection attempts failed (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    |     await get_running_loop().create_connection(
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    |     raise exceptions[0]
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    |     sock = await self._connect_sock(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    |     await self.sock_connect(sock, address)
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    |     return await fut
    |            ^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    |     raise OSError(err, f'Connect call failed {address}')
    | ConnectionRefusedError: [Errno 61] Connect call failed ('::1', 8000, 0, 0)
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    |     await get_running_loop().create_connection(
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    |     raise exceptions[0]
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    |     sock = await self._connect_sock(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    |     await self.sock_connect(sock, address)
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    |     return await fut
    |            ^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    |     raise OSError(err, f'Connect call failed {address}')
    | ConnectionRefusedError: [Errno 61] Connect call failed ('127.0.0.1', 8000)
    +------------------------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 230, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 112, in connect_tcp
    with map_exceptions(exc_map):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/projects/public/localpilot/proxy.py", line 65, in proxy
    r = await client.post(url, data=data, headers=headers, timeout=30)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 365, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
FGarridoV commented 9 months ago

Same issue here.

surak commented 9 months ago

Seems like there should be another server running at port 8000, and it isn't.

surak commented 9 months ago

Okay, I tried running the model call which should open a 8000: 

python3 -m llama_cpp.server --model '/Users/me/models/codellama-7b.Q5_K_S.gguf' --n_gpu_layers 1 --n_ctx 4096

llm_load_tensors: ggml ctx size =    0.09 MB
llm_load_tensors: mem required  = 4435.68 MB
..................................................................................................
llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size  = 2048.00 MB
llama_new_context_with_model: ggml_metal_init() failed
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/surak/Devel/localpilot/venv/lib/python3.11/site-packages/llama_cpp/server/__main__.py", line 96, in <module>
    app = create_app(settings=settings)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/surak/Devel/localpilot/venv/lib/python3.11/site-packages/llama_cpp/server/app.py", line 343, in create_app
    llama = llama_cpp.Llama(
            ^^^^^^^^^^^^^^^^
  File "/Users/surak/Devel/localpilot/venv/lib/python3.11/site-packages/llama_cpp/llama.py", line 377, in __init__
    assert self.ctx is not None
           ^^^^^^^^^^^^^^^^^^^^
AssertionError

So the error in my case is on llama.cpp on my machine.

surak commented 9 months ago

The way to see it is not hide the output of the process call for llama.cpp. Just edit the file proxy.py and change this:

diff --git a/proxy.py b/proxy.py
index 09d488c..177989b 100644
--- a/proxy.py
+++ b/proxy.py
@@ -22,7 +22,7 @@ def start_local_server(model_filename):
            "--n_gpu_layers", "1", "--n_ctx", "4096"]  # TODO: set this more correctly
     logging.debug('Running: %s' % ' '.join(cmd))
     local_server_process = subprocess.Popen(
-        cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+        cmd)#, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

 @app.route('/set_target', methods=['POST'])
surak commented 9 months ago

In my case (intel macbook pro), the error is that there's no "gpu" for it to use, so removing the n_gpu_layers from the llama.cpp call seems to work:

diff --git a/proxy.py b/proxy.py
index 09d488c..1e39b98 100644
--- a/proxy.py
+++ b/proxy.py
@@ -19,7 +19,7 @@ def start_local_server(model_filename):
         local_server_process.terminate()
         local_server_process.wait()
     cmd = ["python3", "-m", "llama_cpp.server", "--model", model_filename,
-           "--n_gpu_layers", "1", "--n_ctx", "4096"]  # TODO: set this more correctly
+        "--n_ctx", "4096"]  # TODO: set this more correctly
     logging.debug('Running: %s' % ' '.join(cmd))
     local_server_process = subprocess.Popen(
         cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
surak commented 9 months ago

Another way of testing your local llama.cpp is just opening http://localhost:8000/docs