Closed foxxxx001 closed 8 months ago
docker build -t llm-api:tgi -f docker/Dockerfile.tgi .
model=/data/checkpoints/SUS-Chat-34B
docker run --gpus=all --shm-size 10g -d -p 7891:80 \
-v /data/checkpoints:/data/checkpoints \
llm-api:tgi --model-id $model --trust-remote-code
docker-compose.yml文件参考如下:
version: '3.10'
services:
apiserver:
image: llm-api:tgi
command: python api/server.py
ulimits:
stack: 67108864
memlock: -1
environment:
- PORT=8000
- MODEL_NAME=sus-chat
- ENGINE=tgi
- TGI_ENDPOINT=http://192.168.20.59:7891 # 第二步TGI启动的IP和端口
volumes:
- $PWD:/workspace
env_file:
- .env.example
ports:
- "7892:8000"
restart: always
networks:
- apinet
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['0']
capabilities: [gpu]
networks:
apinet:
driver: bridge
name: apinet
最后启动转发服务
docker-compose up -d
提交前必须检查以下项目 | The following items must be checked before submission
问题类型 | Type of problem
启动命令 | Startup command
操作系统 | Operating system
None
详细描述问题 | Detailed description of the problem
Dependencies
运行日志或截图 | Runtime logs or screenshots