cocktailpeanut / dalai

The simplest way to run LLaMA on your local machine
https://cocktailpeanut.github.io/dalai
13.09k stars 1.42k forks source link

API samples Notworking on Docker container #393

Open student-ops opened 1 year ago

student-ops commented 1 year ago

I'm woking on the docker provided here and I want to run it as an API. I tried the examples also provided README in the devcontainer, and while the fans are spinning well 🤣 and everything seems to be working fine,but I'm not getting any response. I have installed the model and confirmed that it works in the browser. Can you provide any guidance on how to proceed?

root@9ea6b4db2789:~/dalai# ls 
alpaca  index.js    llama         package-lock.json  server.js  venv
config  install.js  node_modules  package.json       tmp
root@9ea6b4db2789:~/dalai# node server.js
mkdir /root/dalai
> query: {
  url: 'ws://localhost:4000',
  model: 'alpaca.7B',
  prompt: 'The following is a conversation between a boy and a girl:'
}
{ Core: 'alpaca', Model: '7B' }
exec: /root/dalai/alpaca/main --seed -1 --threads 8 --n_predict 128 --model models/7B/ggml-model-q4_0.bin -p "The following is a conversation between a boy and a girl:" in /root/dalai/alpaca

index.js

const Dalai = require("dalai")
new Dalai().request({
  url: "ws://localhost:4000",
  model: "alpaca.7B",
  prompt: "The following is a conversation between a boy and a girl:",
}, (token) => {
  console.log("token", token)
})

server.js

const Dalai = require('dalai')
new Dalai().serve(4000)