Tabby server crashing with sqlite database is locked error.

Mizzlr commented 3 months ago

Describe the bug A clear and concise description of what the bug is.

tabby-1 | The application panicked (crashed). tabby-1 | Message: Must be able to initialize db: while executing migrations: error returned from database: (code: 5) database is locked tabby-1 | tabby-1 | Caused by: tabby-1 | 0: error returned from database: (code: 5) database is locked tabby-1 | 1: (code: 5) database is locked tabby-1 | Location: /root/workspace/ee/tabby-webserver/src/webserver.rs:52 tabby-1 | tabby-1 | Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. tabby-1 | Run with RUST_BACKTRACE=full to include source snippets. tabby-1 exited with code 0

Information about your version Please provide output of tabby --version docker.io/tabbyml/tabby:20240721 Information about your GPU Please provide output of nvidia-smi

+---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 1723 G /usr/lib/xorg/Xorg 90MiB | | 0 N/A N/A 2039 G /usr/bin/gnome-shell 56MiB | +---------------------------------------------------------------------------------------+

Additional context Add any other context about the problem here.

Mizzlr commented 3 months ago

Full backtrace here

Mizzlr commented 3 months ago

Here is the docker compose

  tabby:
    restart: always
    image: tabbyml/tabby:20240721
    pull_policy: always
    # https://tabby.tabbyml.com/docs/models/
    command: serve --model DeepseekCoder-6.7B --device cuda
    volumes:
      - /mnt/disk1/tabby:/data
    ports:
      - 8008:8080
    environment:
      - RUST_BACKTRACE=full
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu

And here is the tabby folder

/mnt/disk1/tabby
├── ee
│   ├── db.sqlite
│   └── db.sqlite-journal
└── models
    └── TabbyML
        ├── DeepseekCoder-6.7B
        │   ├── ggml
        │   │   └── model.gguf
        │   └── tabby.json
        ├── models.json
        └── Nomic-Embed-Text
            ├── ggml
            │   └── model.gguf
            └── tabby.json

7 directories, 7 files

It was working until I decide to upgrade the tabby version to latest (from yesterday). I tried deleting the tabby db files and also tabby folder completely to start fresh. Still the same issue.

wsxiaoys commented 3 months ago

Hi - it seems that your setup have multiple progress accessing the same ~/.tabby/, could you confirm?

Mizzlr commented 3 months ago

I am running tabby server inside docker container of different host and using tabby from pycharm on a different host. This setup was working for me. I could not find any ~/.tabby/ folder on any of my hosts.

host1 --> host2 pycharm with tabby plugin --> tabby server inside docker container no ~/.tabby --> no ~/.tabby

Mizzlr commented 3 months ago

Hi @wsxiaoys Fixed it after adding init: true to docker compose. This is to run docker with proper init process wrapping tabby. In my case, there were zombie processes generated by tabby that were interfering with init process.

Now tabby is working, after some clean restarts

services:
  tabby2:
    restart: always
    image: tabbyml/tabby:20240721
    pull_policy: always
    init: true
    # https://tabby.tabbyml.com/docs/models/
    command: serve --model DeepseekCoder-6.7B --device cuda
    volumes:
      - /mnt/disk1/tabby4:/data
    ports:
      - 8008:8080
    environment:
      - RUST_BACKTRACE=full
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu

⠸   134.738 s   Starting...
tabby-tabby2-1: 
tabby-tabby2-1: ████████╗ █████╗ ██████╗ ██████╗ ██╗   ██╗
tabby-tabby2-1: ╚══██╔══╝██╔══██╗██╔══██╗██╔══██╗╚██╗ ██╔╝
tabby-tabby2-1:    ██║   ███████║██████╔╝██████╔╝ ╚████╔╝ 
tabby-tabby2-1:    ██║   ██╔══██║██╔══██╗██╔══██╗  ╚██╔╝  
tabby-tabby2-1:    ██║   ██║  ██║██████╔╝██████╔╝   ██║   
tabby-tabby2-1:    ╚═╝   ╚═╝  ╚═╝╚═════╝ ╚═════╝    ╚═╝   
tabby-tabby2-1: 
tabby-tabby2-1: 📄 Version 0.13.0-dev.0
tabby-tabby2-1: 🚀 Listening at 0.0.0.0:8080

Unfortunately my history was not backed up and is lost. Will make sure to have back up of the db and events when upgrading next time. Closing this issue.

Mizzlr commented 3 months ago

And thank you for building Tabby, coming from a true stargazer.

TabbyML / tabby

Tabby server crashing with sqlite database is locked error. #2702