Unable to access API on OSX client

marc-w commented 9 months ago

System Info

Python 3.10.11 Requests: 2.31.0 OSX: 13.3.1 GOT4ALL: 2.5.4

Enable API is ON for the application. Application is running and responding.

Information

[X] The official example notebooks/scripts
[X] My own modified scripts

Reproduction

Run Python, 404 returned.

import requests
from pprint import pprint

req = requests.get(
    'http://localhost:4891/docs'
    )

pprint(req.__dict__)

Returns

{'_content': b'',
 '_content_consumed': True,
 '_next': None,
 'connection': <requests.adapters.HTTPAdapter object at 0x1036ed0f0>,
 'cookies': <RequestsCookieJar[]>,
 'elapsed': datetime.timedelta(microseconds=1280),
 'encoding': None,
 'headers': {'Content-Type': 'application/x-empty', 'Access-Control-Allow-Origin': '*', 'Content-Length': '0'},
 'history': [],
 'raw': <urllib3.response.HTTPResponse object at 0x103aea500>,
 'reason': 'Not Found',
 'request': <PreparedRequest [GET]>,
 'status_code': 404,
 'url': 'http://localhost:4891/docs'}

Some of the URLs tried (same response)

http://localhost:4891/docs
http://localhost:4891/
http://localhost:4891/v1
and others I find in the docs
variations with and without trailing slashes

Expected behavior

Return 2xx or similar success.

I have spent 5+ hours reading docs and code plus support issues. This seems to be a feature that exists but does not work. I am not the only one to have issues per my research.

Per a post here:

https://github.com/nomic-ai/gpt4all/issues/1128

This command in bash:

nc -zv 127.0.0.1 4891

returns:

Connection to 127.0.0.1 port 4891 [tcp/*] succeeded!

Hinting at possible success.

I just need to know if the OSX client is capable of local API calls (following the OpenAI spec, something I have had success with previously on OpenAPI service).

If NOT, then I will try a pure python implementation.

If this is a feature, please explain briefly what I am missing and where valid docs exist. (if not I can write them)

Thanks!

dpsalvatierra commented 9 months ago

Since API is running on ubuntu already (containerized) and this setup works across windows and linux distros, I am thinking that perhaps this has to do with how mac osx is binding its dedicated loopback with docker. I will try to test it and see if I can undust ventura.

dpsalvatierra commented 9 months ago

Test with an alias: sudo ifconfig lo0 alias 127.0.0.2 up

then curl http://127.0.0.2:4891/docs#/

dpsalvatierra commented 9 months ago

After digging around, I have confirmed this as being a problem with OSX and Docker desktop. I fired up an old Ventura VM on my esxi server and recreated the problem with a modified version of your script:

import requests
import subprocess
from pprint import pprint

host = input("Enter your hostname: ")

req = requests.get(
    f'http://{host}:4891/docs'
    )

pprint(req.__dict__)

def get_system_info():
        mac_version = subprocess.check_output(["sw_vers"], text=True)
        docker_version = subprocess.check_output(["docker", "--version"], text=True)
        return mac_version, docker_version

def main():
        mac_version, docker_version = get_system_info()
        print("macOSX Version Information:\n", mac_version)
        print("Docker Version Information:\n", docker_version)

if __name__ == "__main__":
        main()

Recreating the problem  #### Socket error: 

(gpt4all) dsalvat1@Daniels-Mac gpt4all % python -m test                                   
Enter your hostname: localhost
Traceback (most recent call last):
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Users/dsalvat1/miniconda3/envs/gpt4all/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer

Wait a few moments, then application starts up 

gpt4all_api  | INFO:     Application startup complete.
gpt4all_api  | INFO:     192.168.65.1:43066 - "GET /docs HTTP/1.1" 200 OK

Results again from the python script: 

(gpt4all) dsalvat1@Daniels-Mac gpt4all % python -m test                    
Enter your hostname: localhost
{'_content': b'\n    <!DOCTYPE html>\n    <html>\n    <head>\n    <link type="t'
             b'ext/css" rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swa'
             b'gger-ui-dist@5.9.0/swagger-ui.css">\n    <link rel="shortcut '
             b'icon" href="https://fastapi.tiangolo.com/img/favicon.png">\n '
             b'   <title>GPT4All API - Swagger UI</title>\n    </head>\n    <'
             b'body>\n    <div id="swagger-ui">\n    </div>\n    <script src="'
             b'https://cdn.jsdelivr.net/npm/swagger-ui-dist@5.9.0/swagger-ui-bu'
             b'ndle.js"></script>\n    <!-- `SwaggerUIBundle` is now availab'
             b'le on the page -->\n    <script>\n    const ui = SwaggerUIBund'
             b'le({\n        url: \'/openapi.json\',\n    "dom_id": "#swagger-u'
             b'i",\n"layout": "BaseLayout",\n"deepLinking": true,\n"showExtens'
             b'ions": true,\n"showCommonExtensions": true,\noauth2RedirectUrl'
             b": window.location.origin + '/docs/oauth2-redirect',\n    pres"
             b'ets: [\n        SwaggerUIBundle.presets.apis,\n        Swagger'
             b'UIBundle.SwaggerUIStandalonePreset\n        ],\n    })\n    </s'
             b'cript>\n    </body>\n    </html>\n    ',
 '_content_consumed': True,
 '_next': None,
 'connection': <requests.adapters.HTTPAdapter object at 0x7f9ae82b2260>,
 'cookies': <RequestsCookieJar[]>,
 'elapsed': datetime.timedelta(microseconds=4511),
 'encoding': 'utf-8',
 'headers': {'date': 'Sun, 10 Dec 2023 18:19:49 GMT', 'server': 'uvicorn', 'content-length': '943', 'content-type': 'text/html; charset=utf-8'},
 'history': [],
 'raw': <urllib3.response.HTTPResponse object at 0x7f9ae834ba00>,
 'reason': 'OK',
 'request': <PreparedRequest [GET]>,
 'status_code': 200,
 'url': 'http://localhost:4891/docs'}
macOSX Version Information:
 ProductName:   macOS
ProductVersion: 13.0
BuildVersion:   22A5266r

Docker Version Information:
Docker version 24.0.7, build afdd53b

Docker compose V3.8 - legacy docker loopback

Also noticed that when running docker compose, I am not able to ping host.docker.internal but a legacy localhost called “docker.for.mac.localhost” 

output with curl command: 

(gpt4all) dsalvat1@Daniels-Mac gpt4all % curl http://docker.for.mac.localhost:4891/docs\#/

    <!DOCTYPE html>
    <html>
    <head>
    <link type="text/css" rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5.9.0/swagger-ui.css">
    <link rel="shortcut icon" href="https://fastapi.tiangolo.com/img/favicon.png">
    <title>GPT4All API - Swagger UI</title>

Potential issue:

The mac setup takes a minute to serve the inference on the LLM and perhaps this may be a permission issue specific to mac, can you post the logs generated when you start the container?

Here are mine

gpt4all_api  | [2023-12-10 19:24:29,377 7:MainThread] main - INFO - Downloading/fetching model: /models/mistral-7b-instruct-v0.1.Q4_0 | main.py:37
gpt4all_api  | [2023-12-10 19:25:26,968 7:MainThread] main - INFO - GPT4All API is ready to infer from mistral-7b-instruct-v0.1.Q4_0 on CPU. | main.py:42

Almost a minute to load, the already downloaded language model is available and after that, both the curl and python get calls back from the requests.

marc-w commented 9 months ago

Great work @dpsalvatierra - trying to keep your order:

Note: I updated OSX over the weekend, so system info is different from initial report.

mac osx is binding its dedicated loopback with docker

That is possible. I don't work with Docker directly anymore, I am a little out of the loop. In general Docker is very problematic on OSX, has been that way since 2014 for me. Ergo...

sudo ifconfig lo0 alias 127.0.0.2 up

✅

then curl curl http://127.0.0.2:4891/docs\#/

App is running, has been running for minutes.

(GPT4ALL) tilt@tilt GPT4ALL % curl http://127.0.0.2:4891/docs\#/
curl: (7) Failed to connect to 127.0.0.2 port 4891 after 7 ms: Couldn't connect to server

Below is the result of the Python posted, still getting that 404.

I also waited a few minutes after GPT4All is running and ran the Python again with both localhost and '127.0.0.1' - 404 on both.

I use PIPENV for virtualization.

(GPT4ALL) tilt@tilt GPT4ALL % python main.py 
Enter your hostname: localhost
{'_content': b'',
 '_content_consumed': True,
 '_next': None,
 'connection': <requests.adapters.HTTPAdapter object at 0x10533a9e0>,
 'cookies': <RequestsCookieJar[]>,
 'elapsed': datetime.timedelta(microseconds=26237),
 'encoding': None,
 'headers': {'Content-Type': 'application/x-empty', 'Access-Control-Allow-Origin': '*', 'Content-Length': '0'},
 'history': [],
 'raw': <urllib3.response.HTTPResponse object at 0x10541c1f0>,
 'reason': 'Not Found',
 'request': <PreparedRequest [GET]>,
 'status_code': 404,
 'url': 'http://localhost:4891/docs'}
macOSX Version Information:
 ProductName:       macOS
ProductVersion:     14.1.2
BuildVersion:       23B92

Docker Version Information:
 Docker version 20.10.17, build 100c701

Using http://docker.for.mac.localhost:4891/docs\#/

returns nothing, tried a few variations.
nothing via curl http://host.docker.internal.localhost:4891/docs

For the hell of it I updated Docker (via Docker desktop app) to v4.26.0 - same results. Just incase my old Docker had pooled resources messing us up.

Regarding gpt4all_api - and logs - preferred way to get that from the OSX app? To be clear, I am not running Docker, only the app that in turn uses Docker.

Happy to help, looks like more OSX/Docker shenanigans.

May be important, none of the Python/command line work returned anything close to a correct answer. I went through most of the Python bindings and other work before writing this ticket. If I could run this terminal only that would be fine.

Right now, the OSX app runs as expected and that would be perfect for some local training and interface building.

Kudos for allowing me to work locally and have statefulness.

dpsalvatierra commented 9 months ago

Very briefly, can you post the output for docker compose up that shows the LLM being loaded during the deployment of the container.

marc-w commented 9 months ago

I can filter down to gpt4all from OSX console app.

Otherwise show me an easy way to grab the logs (console tail = ?) from the app. No logs (or app listed) in /Library/Application Support.

dpsalvatierra commented 9 months ago

In console: docker logs gpt4all_api

marc-w commented 9 months ago

✅

(GPT4ALL) tilt@tilt GPT4ALL % docker logs gpt4all_api
Error response from daemon: No such container: gpt4all_api

From console when I run the command:

default 14:03:45.296570-0700    com.docker.backend  proxy >> GET /v1.43/containers/gpt4all_api/json
default 14:03:45.296612-0700    com.docker.backend  idle -> busy (was idle for 53.086µs; reason: [in-flight APIs: map[/v1.43/containers/gpt4all_api/json:1]])
default 14:03:45.298705-0700    com.docker.backend  proxy << GET /v1.43/containers/gpt4all_api/json (2.132299ms)

dpsalvatierra commented 9 months ago

Container needs to be running, can you start it up in console? cd to the gpt4all api folder and execute "docker compose up" then paste the output here.

Folder should be where you checked out the github files "~/gpt4all/gpt4all-api"

marc-w commented 9 months ago

I appreciate the support; I don't think we are on the same page.

Again, I am using the OSX app. I am playing dumb. In the app, there is a setting for "enable API," which is checked. The API is not included in the app; that is fine.

So, I did not download the repo; I downloaded the OSX app, installed it, and opened it.

It is important to note that not everyone likes, uses, or even thinks Docker is a good idea.

I hope this issue finds others who may be wasting their time or employer money trying to get this working.

Corrections welcome.

cebtenzzre commented 9 months ago

Reopening as it sounds like there is still an issue with the chat UI's API server.

marc-w commented 8 months ago

There is; I closed it for obvious reasons. Love to see it fixed, here to help.

nomic-ai / gpt4all