search

huggingface / text-embeddings-inference

A blazing fast inference solution for text embeddings models
https://huggingface.co/docs/text-embeddings-inference/quick_tour
Apache License 2.0
2.64k stars 163 forks source link

support build gpu image with lower cuda version #181

Open DeoLeung opened 6 months ago

DeoLeung commented 6 months ago

Feature request

not sure if the current cuda 12.2 is a hard limit or could be optional

if optional, could parametrize the docker base image so user could re-build it easily

Motivation

some server just hard to upgrade the cuda version

Your contribution

parametrized the dockerfile and rebuild it if necessary on users' end

supdizh commented 5 months ago

+1 for this, why TEI request cuda12.2, while torch is still with cuda12.1, and there is no nv-driver compatible for both 12.1/12.2. https://docs.nvidia.com/deploy/cuda-compatibility/index.html

OlivierDehaene commented 5 months ago

and there is no nv-driver compatible for both 12.1/12.2

From the page you linked:

If you are upgrading the driver to 525.60.13 which is the minimum required driver version for the 12.x toolkits, then 11.x and 12.x applications will be supported due to backward compatibility and future 12.x applications will be supported due to minor-version compatibility.

You can run both torch and TEI with >=525.60.13.

@DeoLeung do you have issues running TEI on a server with >=525.60.13?

DeoLeung commented 5 months ago

and there is no nv-driver compatible for both 12.1/12.2

From the page you linked:

If you are upgrading the driver to 525.60.13 which is the minimum required driver version for the 12.x toolkits, then 11.x and 12.x applications will be supported due to backward compatibility and future 12.x applications will be supported due to minor-version compatibility.

You can run both torch and TEI with >=525.60.13.

@DeoLeung do you have issues running TEI on a server with >=525.60.13?

we managed to do a server upgrade and it's now running fine on NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4

supdizh commented 5 months ago

and there is no nv-driver compatible for both 12.1/12.2

From the page you linked:

If you are upgrading the driver to 525.60.13 which is the minimum required driver version for the 12.x toolkits, then 11.x and 12.x applications will be supported due to backward compatibility and future 12.x applications will be supported due to minor-version compatibility.

You can run both torch and TEI with >=525.60.13.

@DeoLeung do you have issues running TEI on a server with >=525.60.13?

othx, maybe I should try downgrade to 525.60.13 I'm with 530.30.02 now, it seems not compatible from the table in the link starting the tei docker will report a error : @OlivierDehaene

nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.2, please update your driver to a newer version, or use an earlier cuda container: unknown.