search

cocktailpeanut / dalai

The simplest way to run LLaMA on your local machine
https://cocktailpeanut.github.io/dalai
13.09k stars 1.42k forks source link

`npx dalai serve` not showing any models in its web page #277

Open meidaid opened 1 year ago

meidaid commented 1 year ago

My environment: Ubuntu Linux 22.04 LTS.

I can run npx dalai serve, however no models show up in its web page.

When I run npx dalai server I get the following output:

$ npx dalai serve
mkdir /home/xxxx/dalai
Server running on http://localhost:3000/
> query: { method: 'installed', models: [] }
modelsPath /home/xxxx/dalai/alpaca/models
{ modelFolders: [ '7B' ] }
modelsPath /home/xxxx/dalai/llama/models
{ modelFolders: [ '13B', '7B' ] }

The webpage looks like the following:

Screenshot from 2023-03-26 13-16-02

Notice that the model's drop down selector is empty.

The last background that I can offer is that I manually downloaded the llama models and alpaca model.

Due to disk space constraints, only llama 7B is physically in the llama models directory. llama 13B and alpaca 7B are referenced using soft links to external storage.

I feel like I am so close here. Any ideas?

mapleroyal commented 1 year ago

OK I worked it out:

The models need to be compatible models. I don't know how it determines it, but that's that. I had struggled so much and tried so many different installs, that on the most recent one I forgot to quantize them, which meant it couldn't see any compatible models.

However, even after using the true original facebook models, converting them with no issues, quantizing them with no issues, and putting them where dalai can see them, then selecting each one individually one after the other making sure to clear site data and cookies and close the browser and quit dalai and reopen dalai between each model test, it still fails to produce any output. My CPUs show no activity at all. Something is wrong :(

I tried this in several VMs and 2 different physical machines.

NIPSEN commented 1 year ago

I had the same issue. I downloaded model 13B from a torrent.

mapleroyal commented 1 year ago

Updated my comment above with newer info.

HoneyCodeBadger commented 1 year ago

Same issue here. No clue why.

` C:\Users\User>npx dalai serve mkdir C:\Users\User\dalai Server running on http://localhost:3000/

query: { method: 'installed', models: [] } modelsPath C:\Users\User\dalai\alpaca\models { modelFolders: [] } modelsPath C:\Users\User\dalai\llama\models { modelFolders: [ '65B', '7B' ] } `

HoneyCodeBadger commented 1 year ago

Do this:

npx dalai alpaca install 7B

Then you can use alpaca.

meidaid commented 1 year ago

OK I worked it out:

The models need to be compatible models. I don't know how it determines it, but that's that. I had struggled so much and tried so many different installs, that on the most recent one I forgot to quantize them, which meant it couldn't see any compatible models.

  • clear the cookies and site data for the dalai page (click the lock to the left of the address bar)
  • there can be other stuff in the folders, confirmed
  • the files can be symlinked, confirmed
  • they have to be in one of these locations:
  • /home/username/dalai/llama/models/7B or /home/username/dalai/alpaca/models/7B (obviously use appropriate folder names for the models, 7B, 13B, 30B, and 65B).

However, even after using the true original facebook models, converting them with no issues, quantizing them with no issues, and putting them where dalai can see them, then selecting each one individually one after the other making sure to clear site data and cookies and close the browser and quit dalai and reopen dalai between each model test, it still fails to produce any output. My CPUs show no activity at all. Something is wrong :(

I tried this in several VMs and 2 different physical machines.

Thank you for the respose. I've rolled back the VM environment and tried running npx dalai serve again. As for the browser, I am using secret mode so that every time I start the environment is clean:

$ npx dalai serve
mkdir /home/david/dalai
Server running on http://localhost:3000/
> query: { method: 'installed', models: [] }
modelsPath /home/david/dalai/alpaca/models
{ modelFolders: [ '7B' ] }
modelsPath /home/david/dalai/llama/models
{ modelFolders: [ '13B', '7B' ] }

I am using symbolic links. But I still get an empty drop down.

I am purging out my install and will redo the install fresh.

mapleroyal commented 1 year ago

A few thoughts, because I've got it to work consistently now on 3 machines (two bare metal, 1 KVM):

meidaid commented 1 year ago

Thanks for the reply!

I am starting to wonder if I performed a step wrong. I downloaded the models directly from their source site using the wget command. Perhaps I should have instead used the npx dalai install ... commands, instead. Perhaps that command does some kind of post processing in the environment that makes it so that the model files become visible to npx dalai serve?

NotendoAtendo commented 1 year ago

Thanks for the reply!

I am starting to wonder if I performed a step wrong. I downloaded the models directly from their source site using the wget command. Perhaps I should have instead used the npx dalai install ... commands, instead. Perhaps that command does some kind of post processing in the environment that makes it so that the model files become visible to npx dalai serve?

Just so you know, I went the npx dalai install route and the only model I got to work was alpaca, the others dont show up at all. The consol states they are installed but the dropdown stays empty, just like yours.

Lakeshire-Empress commented 1 year ago

After I'm in the interface, I can't seem to find models anywhere, I could still type things in but there was never response in return.

Lakeshire-Empress commented 1 year ago

cleaning up temp filesined python 100%[====================================================================>] done exec: pip3 install --user virtualenv in undefined} innerError Error: Cannot find module '../build/Debug/conpty.node' Require stack:

C:\WINDOWS\system32>npx dalai alpaca install 7B mkdir C:\Users\varia\dalai { method: 'install', callparams: [ '7B' ] } extracting pythoncontained python 100%[====================================================================>] done cleaning up temp files exec: pip3 install --user virtualenv in undefined innerError Error: Cannot find module '../build/Debug/conpty.node' Require stack:

extracting pythoncontained python 0% / cleaning up temp filesined python 100%[====================================================================>] done exec: pip3 install --user virtualenv in undefined innerError Error: Cannot find module '../build/Debug/conpty.node' Require stack:

C:\WINDOWS\system32>npm install -g node-gyp npm WARN deprecated @npmcli/move-file@2.0.1: This functionality has been moved to @npmcli/fs

added 95 packages in 4s

5 packages are looking for funding run npm fund for details

C:\WINDOWS\system32>npm install -g windows-build-tools npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated windows-build-tools@5.2.2: Node.js now includes build tools for Windows. You probably no longer need this tool. See https://github.com/felixrieseberg/windows-build-tools for details. npm WARN cleanup Failed to remove some directories [ npm WARN cleanup [ npm WARN cleanup 'C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools', npm WARN cleanup [Error: EPERM: operation not permitted, rmdir 'C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\node_modules\verror\node_modules'] { npm WARN cleanup errno: -4048, npm WARN cleanup code: 'EPERM', npm WARN cleanup syscall: 'rmdir', npm WARN cleanup path: 'C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\node_modules\verror\node_modules' npm WARN cleanup } npm WARN cleanup ] npm WARN cleanup ] npm ERR! code 1 npm ERR! path C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools npm ERR! command failed npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node ./dist/index.js Downloading python-2.7.15.amd64.msi npm ERR! [============================================>] 100.0% of 20.2 MB (20.2 MB/s) npm ERR! Downloaded python-2.7.15.amd64.msi. Saved to C:\Users\varia.windows-build-tools\python-2.7.15.amd64.msi. Downloading vs_BuildTools.exe npm ERR! [============================================>] 100.0% of 1.12 MB (1.12 MB/s) npm ERR! Downloaded vs_BuildTools.exe. Saved to C:\Users\varia.windows-build-tools\vs_BuildTools.exe. npm ERR! npm ERR! Starting installation... npm ERR! Downloading installers failed. Error: TypeError: 'process.env' only accepts a configurable, writable, and enumerable data descriptor npm ERR! at Function.defineProperty () npm ERR! at Object.removePath (C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\dist\utils\remove-path.js:11:12) npm ERR! at Object.install (C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\dist\install\index.js:29:19) npm ERR! at C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\dist\start.js:17:19 npm ERR! at Object.download (C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\dist\download.js:35:5) npm ERR! at process.processTicksAndRejections (node:internal/process/task_queues:95:5) npm ERR! at async Object.aquireInstallers (C:\Users\varia\AppData\Roaming\npm\node_modules\windows-build-tools\dist\aquire-installers.js:32:13) { npm ERR! code: 'ERR_INVALID_OBJECT_DEFINE_PROPERTY' npm ERR! } npm ERR! windows-build-tools will now exit.

npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\varia\AppData\Local\npm-cache_logs\2023-05-23T20_19_55_138Z-debug-0.log

C:\WINDOWS\system32>npx dalai clean ############################################ #

Supported Commands:

#

1. System command

#

dalai serve <port (optional)>

#

2. Model command

#

dalai llama get

# ############################################

C:\WINDOWS\system32>npx dalai alpaca install 7B mkdir C:\Users\varia\dalai { method: 'install', callparams: [ '7B' ] } extracting pythoncontained python 0% / cleaning up temp filesined python 100%[====================================================================>] done exec: pip3 install --user virtualenv in undefined innerError Error: Cannot find module '../build/Debug/conpty.node' Require stack:

C:\WINDOWS\system32>npx dalai serve mkdir C:\Users\varia\dalai Server running on http://localhost:3000/

query: { method: 'installed', models: [] } modelsPath C:\Users\varia\dalai\alpaca\models { modelFolders: [] } modelsPath C:\Users\varia\dalai\llama\models { modelFolders: [] }

MohsenShabanian commented 1 year ago

Very informative, thank you all. I had the same problem on an ubuntu sandbox and the install command worked after reading this article... I don't know why but here I am.

@meidaid a question for you please. I want to separate my work from my sandbox, by running Ubuntu off a USB key and put the data and models on a NAS folder or another USB key. This way I can upgrade the hardware and processing if I like playing with AI. Does that make sense to you? Also all I have to do after each potential wipe and reinstall is to create symbolic links to the models in /home/username/dalai/alpaca/models/7B? or can I just link to the folder?

meidaid commented 1 year ago

After that post I found that my soft linking strategy did not work. It seemed that I soft linked the large model files instead of a directory or something like that.

Maybe if I used the NAS approach you are proposing it would've worked as the directory would've been consistent.

As for using USB sticks for the Ubuntu environment: Your approach sounds clever.

MohsenShabanian commented 1 year ago

Thanks for responding.

On Wed., Jul. 12, 2023, 9:39 p.m. meidaid, @.***> wrote:

After that post I found that my soft linking strategy did not work. It seemed that I soft linked the large model files instead of a directory or something like that.

Maybe if I used the NAS approach you are proposing it would've worked as the directory would've been consistent.

As for using USB sticks for the Ubuntu environment: Your approach sounds clever.

— Reply to this email directly, view it on GitHub https://github.com/cocktailpeanut/dalai/issues/277#issuecomment-1633411385, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWAPE32OJ75NUI5APK5RLTXP5GVRANCNFSM6AAAAAAWH6CK6Q . You are receiving this because you commented.Message ID: @.***>