This project consists of a frontend application and a backend setup using Docker.
Before you begin, ensure you have the following installed on your machine:
Navigate to the root project directory:
cd /path/to/project
Install dependencies:
npm install
Update the .env.local
file with the following configurations:
OPENAI_BASE_URL=http://localhost:5000/v1/
TOKENIZE_BASE_URL=http://localhost:3348
TTS_BASE_URL=http://localhost:22311/v1/
Build the project:
npm run build
Start the frontend:
npm start
Navigate to the Docker folder:
cd docker
Build the Docker containers:
docker-compose build
Download the latest Llama3s model in ExLlama2 format from: Llama3-s-instruct-v0.3-checkpoint-7000-phase-3-exllama2
Edit the docker/tabbyapi/config.yml
file:
model_name:
field with the folder path containing the Llama3s ExLlama2 model.Update the Docker Compose configuration:
In your docker-compose.yml
file, ensure the tabbyapi
service has the following configuration:
tabbyapi:
container_name: tabbyapi
build:
context: ./tabbyAPI-personal-fork
dockerfile: ./docker/Dockerfile
args:
DO_PULL: "true"
ports:
- "5000:5000"
environment:
NAME: TabbyAPI
NVIDIA_VISIBLE_DEVICES: all
volumes:
- /path/to/parent/directory/of/llama3s/:/app/models
- ./tabbyapi/config.yml:/app/config.yml
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [ gpu ]
Make sure to replace /path/to/parent/directory/of/llama3s/
with the actual path to the parent directory containing your Llama3s model.
Start the Docker containers:
docker-compose up
Once everything is set up and running, you can access the demo page by opening your browser and navigating to:
http://localhost:3000
If you encounter any issues during the setup or running of the project, please check the following:
If problems persist, please open an issue in this repository with detailed information about the error you're experiencing.