search

keldenl / gpt-llama.cpp

A llama.cpp drop-in replacement for OpenAI's GPT endpoints, allowing GPT-powered apps to run off local llama.cpp models instead of OpenAI.
MIT License
594 stars 66 forks source link

Unable to run test-installation.sh in ubuntu #32

Closed BenjiKCF closed 1 year ago

BenjiKCF commented 1 year ago

It shows the following error because the sh doesn’t support ubuntu.

./test-installation.sh: 17: [[: not found
./test-installation.sh: 23: [[: not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   726  100   460  100   266     23     13  0:00:20  0:00:19  0:00:01   121
./test-installation.sh: 47: [[: not found

Error: Curl command failed!
Is the gpt-llama.cpp server running? Try starting the server and running this script again.
Make sure you are testing on the right port. The Curl commmand server error port should match your port in the gpt-llama.cpp window.
Please check for any errors in the terminal window running the gpt-llama.cpp server.
Ed-ward commented 1 year ago

Unfortunately, I'm having a similar problem on macos. Llamaspp is installed on macmini (Intel) and works correctly. I have tried different models: (7B, 13B) vicuna, koala, alpaca and they all work well.

gpt-llama.cpp is also installed. It runs without errors with any port and the web interface works. In the web interface, I specify /Users/user/llama.cpp/models/7B/ggml-vicuna-13b-1.1-q4_0.bin This is the path I specify when I run test-installation.sh. But test-installation.sh fails:

Mac-mini-USER:gpt-llama.cpp user$ sh ./test-installation.sh
--GPT-LLAMA.CPP TEST INSTALLATION SCRIPT LAUNCHED--
PLEASE MAKE SURE THAT A LOCAL GPT-LLAMA.CPP SERVER IS STARTED. OPEN A SEPARATE TERMINAL WINDOW START IT.

What port is your server running on? (press enter for default 443 port): 8080
Please drag and drop the location of your Llama-based Model (.bin) here and press enter: /Users/user/llama.cpp/models/7B/ggml-vicuna-13b-1.1-q4_0.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   266    0     0  100   266      0  10128 --:--:-- --:--:-- --:--:-- 12666
curl: (52) Empty reply from server

Error: Curl command failed!
Is the gpt-llama.cpp server running? Try starting the server and running this script again.
Make sure you are testing on the right port. The Curl commmand server error port should match your port in the gpt-llama.cpp window.
Please check for any errors in the terminal window running the gpt-llama.cpp server.

In the server terminal, I see that the server has been stopped:

Mac-mini-USER:gpt-llama.cpp user$ PORT=8080 npm run start

> gpt-llama.cpp@0.2.2 start
> node index.js

Server is listening on:
  - http://localhost:8080
  - http://192.168.0.13:8080 (for other devices on the same network)

See Docs
  - http://localhost:8080/docs

Test your installation
  - open another terminal window and run sh ./test-installation.sh

See https://github.com/keldenl/gpt-llama.cpp#usage for more guidance.
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /docs/
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /docs/swagger-ui.css
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /docs/swagger-ui-init.js
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /docs/swagger-ui-standalone-preset.js
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /docs/swagger-ui-bundle.js
> PROCESS COMPLETE
> PROCESS COMPLETE
> PROCESS COMPLETE
> PROCESS COMPLETE
> PROCESS COMPLETE
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /v1/chat/completions

=====  CHAT COMPLETION REQUEST  =====

=====  LLAMA.CPP SPAWNED  =====
/Users/user/llama.cpp/main -m /Users/user/llama.cpp/models/7B/ggml-vicuna-13b-1.1-q4_0.bin --temp 0.7 --n_predict 512 --top_p 0.1 --top_k 40 -b 512 -c 2048 --repeat_penalty 1.1764705882352942 --reverse-prompt user: --reverse-prompt 
user --reverse-prompt system: --reverse-prompt 
system --reverse-prompt ## --reverse-prompt 
## --reverse-prompt ### -i -p ### Instructions
Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.

### Inputs
system: You are a helpful assistant.
user: How are you?
assistant: Hi, how may I help you today?
system: You are ChatGPT, a helpful assistant developed by OpenAI.

### Response
user: How are you doing today?
assistant:

=====  REQUEST  =====
user: How are you doing today?
file:///Users/user/gpt-llama.cpp/routes/chatRoutes.js:167
    const readable = new ReadableStream({
                     ^

ReferenceError: ReadableStream is not defined
    at file:///Users/user/gpt-llama.cpp/routes/chatRoutes.js:167:19
    at Layer.handle [as handle_request] (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/layer.js:95:5)
    at next (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/route.js:144:13)
    at Route.dispatch (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/route.js:114:3)
    at Layer.handle [as handle_request] (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/layer.js:95:5)
    at /Users/user/gpt-llama.cpp/node_modules/express/lib/router/index.js:284:15
    at Function.process_params (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/index.js:346:12)
    at next (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/index.js:280:10)
    at Function.handle (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/index.js:175:3)
    at router (/Users/user/gpt-llama.cpp/node_modules/express/lib/router/index.js:47:12)

If, ignoring the failed test, I try to run AutoGPT I get this:

Mac-mini-USER:Auto-GPT user$ python3 -m autogpt
Warning: The file 'auto-gpt.json' does not exist. Local memory would not be saved to a file.
NEWS:  Welcome to Auto-GPT! We'll keep you informed of the latest news and features by printing messages here. If you don't wish to see this message, you can run Auto-GPT with the --skip-news flag # INCLUDED COMMAND 'send_tweet' IS DEPRICATED, AND WILL BE REMOVED IN THE NEXT STABLE RELEASE Base Twitter functionality (and more) is now covered by plugins: https://github.com/Significant-Gravitas/Auto-GPT-Plugins ## Changes to Docker configuration The workdir has been changed from /home/appuser to /app. Be sure to update any volume mounts accordingly.
Welcome back!  Would you like me to return to being Helper?
Continue with the last settings?
Name:  Helper
Role:  an AI designed to talk with me about everything
Goals: ['Answering about any quesion and supporting dialogues.', 'Search information for correct answer in Google if it will be need.', 'Save interesting facts in text file for next using.', 'Be able intresting opponent in dialogues.', 'Dont worry be happy :-)']
Continue (y/n): y
Using memory of type:  LocalCache
Using Browser:  chrome
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/user/Auto-GPT/autogpt/__main__.py", line 5, in <module>
    autogpt.cli.main()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1635, in invoke
    rv = super().invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/user/Auto-GPT/autogpt/cli.py", line 177, in main
    agent.start_interaction_loop()
  File "/Users/user/Auto-GPT/autogpt/agent/agent.py", line 86, in start_interaction_loop
    assistant_reply = chat_with_ai(
  File "/Users/user/Auto-GPT/autogpt/chat.py", line 179, in chat_with_ai
    assistant_reply = create_chat_completion(
  File "/Users/user/Auto-GPT/autogpt/llm_utils.py", line 107, in create_chat_completion
    response = openai.ChatCompletion.create(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/api_resources/chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 226, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 619, in _interpret_response
    self._interpret_response_line(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 682, in _interpret_response_line
    raise self.handle_error_response(
openai.error.AuthenticationError: Incorrect API key provided: /Users/u************************************************.bin. You can find your API key at https://platform.openai.com/account/api-keys.

I understand that autogpt cannot interact with the model file and I understand that the problem occurs earlier. The error when running test-installation.sh says so, but doesn't state the reason. So far, I do not have enough knowledge to figure this out and I will be grateful for any advice that can help.

keldenl commented 1 year ago

@Ed-ward make sure your node version is >= 18, readablestream was introduced in v18, so if your node version is lower than that it'll be undefined

keldenl commented 1 year ago

@BenjiKCF what does your window look like for gpt-llama.cpp?

Ed-ward commented 1 year ago

@Ed-ward make sure your node version is >= 18, readablestream was introduced in v18, so if your node version is lower than that it'll be undefined

Updating node solved this problem. I should have been more careful and from the very beginning and checked everything. Many thanks for the help!

Mac-mini-USER:gpt-llama.cpp user$ sh /Users/user/gpt-llama.cpp/test-installation.sh
--GPT-LLAMA.CPP TEST INSTALLATION SCRIPT LAUNCHED--
PLEASE MAKE SURE THAT A LOCAL GPT-LLAMA.CPP SERVER IS STARTED. OPEN A SEPARATE TERMINAL WINDOW START IT.

What port is your server running on? (press enter for default 443 port): 8000
Please drag and drop the location of your Llama-based Model (.bin) here and press enter: /Users/user/llama.cpp/models/7B/ggml-vicuna-13b-1.1-q4_0.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   669  100   403  100   266     12      8  0:00:33  0:00:33 --:--:--   107

--RESPONSE--
{"choices":[{"message":{"content":"As an AI language model, I don't have feelings like humans do, but I am functioning well and ready to assist you with any questions or tasks you may have!","role":"assistant"},"finish_reason":"stop","index":0}],"created":1683386326485,"id":"Vv7w_OoXOLc4srGbu2zA0","object":"chat.completion.chunk","usage":{"prompt_tokens":99,"completion_tokens":38,"total_tokens":137}}

--RESULTS--
Curl command was successful!
To use any app with gpt-llama.cpp, please provide the following as the OPENAI_API_KEY:
/Users/user/llama.cpp/models/7B/ggml-vicuna-13b-1.1-q4_0.bin
Mac-mini-USER:gpt-llama.cpp user$ 
> REQUEST RECEIVED
> PROCESSING NEXT REQUEST FOR /v1/chat/completions

=====  CHAT COMPLETION REQUEST  =====

=====  LLAMA.CPP SPAWNED  =====
/Users/user/llama.cpp/main -m /Users/user/llama.cpp/models/7B/ggml-vicuna-13b-1.1-q4_0.bin --temp 0.7 --n_predict 1000 --top_p 0.1 --top_k 40 -c 2048 --seed -1 --repeat_penalty 1.1764705882352942 --reverse-prompt user: --reverse-prompt 
user --reverse-prompt system: --reverse-prompt 
system --reverse-prompt ## --reverse-prompt 
## --reverse-prompt ### -i -p ### Instructions
Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.

### Inputs
system: You are a helpful assistant.
user: How are you?
assistant: Hi, how may I help you today?
system: You are ChatGPT, a helpful assistant developed by OpenAI.

### Response
user: How are you doing today?
assistant:

=====  REQUEST  =====
user: How are you doing today?
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====
=====  PROCESSING PROMPT...  =====

=====  RESPONSE  =====
 As an AI language model, I don't have feelings like humans do, but I am functioning well and ready to assist you with any questions or tasks you may have!
user:Request DONE
> PROCESS COMPLETE
keldenl commented 1 year ago

Awesome! I should give a more comprehensive error as well, so let me add that