Open anneum opened 10 months ago
Conda inside docker container? 🤔 For what?
I think the model is better to mount out, how should I start
@storuky I could not get cuda 12 to stay installed without conda because during the installation of ochat, for whatever reason, cuda 11 was installed.
@xiaocode337317439 In my environment, the container is running in a Kubernetes cluster. Here is the corresponding YAML file where the model is stored on a pvc:
apiVersion: apps/v1
kind: Deployment
metadata:
name: openchat-api
namespace: openchat
spec:
selector:
matchLabels:
app: openchat-api
template:
metadata:
labels:
app: openchat-api
spec:
containers:
- name: openchat-api
imagePullPolicy: Always
image: <image path>
resources:
limits:
cpu: "32"
memory: "250Gi"
nvidia.com/gpu: 1
requests:
cpu: "0.5"
memory: "10Gi"
ports:
- containerPort: 18888
protocol: TCP
name: api-port
volumeMounts:
# Persist model data
- mountPath: /root/.cache/huggingface/hub
name: models
volumes:
- name: models
persistentVolumeClaim:
claimName: models
I wanted to share my experience of successfully building a Dockerfile to deploy the openchat API as a container. This process came with some challenges, especially maintaining a stable conda environment during the Docker build, which I had to use, otherwise I didn't get the right versions and dependencies fixed.
I believe that my Dockerfile can simplify many things for users who want to use the openchat project.
The container is running on a host with CUDA version: 12.2.