GPU required even for CPU-only based inference session

[jcntrl]

Exactly (but with slight modification device='cpu' per your example found here and here:

from hf_hub_ctranslate2 import CT2SentenceTransformer
import torch
import sentence_transformers

model_name_pytorch = "intfloat/e5-small"
model = CT2SentenceTransformer(
    model_name_pytorch, compute_type="int8", device="cpu", 
)
embeddings = model.encode(
    ["I like soccer", "I like tennis", "The eiffel tower is in Paris"],
    batch_size=32,
    convert_to_numpy=True,
    normalize_embeddings=True,
)
print(embeddings.shape, embeddings)
scores = (embeddings @ embeddings.T) * 100`

Results in this confusing error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-75-18d6db40833a> in <cell line: 7>()
      5 
      6 model_name_pytorch = "intfloat/e5-small"
----> 7 model = CT2SentenceTransformer(
      8     model_name_pytorch, compute_type="int8", device="cpu",
      9 )

/usr/local/lib/python3.10/dist-packages/hf_hub_ctranslate2/ct2_sentence_transformers.py in __init__(self, compute_type, force, vmap, *args, **kwargs)
     51     ):
     52         if not IS_SentenceTransformer_IMPORTED:
---> 53             raise ValueError("Installation requires sentence_transformers. Please `pip install hf_hub_ctranslate2[sentence_transformers]`")
     54         if not IS_torch_IMPORTED:
     55             raise ValueError("Installation requires torch. Please `pip install hf_hub_ctranslate2[sentence_transformers]`")

ValueError: Installation requires sentence_transformers. Please `pip install hf_hub_ctranslate2[sentence_transformers]`

I have previously installed and verified the installation of these packages: !pip install hf-hub-ctranslate2 hf_hub_ctranslate2[sentence_transformers] torch sentence-transformers

Elsewhere in my notebook, I have installed and can assure torch, sentence_transformers, ctranslate2, transformers, numpy are all present and functioning as expected.

Compute environment is a google colab, cpu-based without a GPU. This error disappears when a T4 is available, even when device='cpu' is specified. The following output seems to be the expected result:

Downloading (…)da68f/.gitattributes: 100%
1.48k/1.48k [00:00<00:00, 58.5kB/s]
Downloading (…)_Pooling/config.json: 100%
200/200 [00:00<00:00, 4.73kB/s]
Downloading (…)662abda68f/README.md: 100%
68.0k/68.0k [00:00<00:00, 830kB/s]
Downloading (…)2abda68f/config.json: 100%
641/641 [00:00<00:00, 9.71kB/s]
Downloading model.onnx: 100%
133M/133M [00:03<00:00, 48.3MB/s]
Downloading model.safetensors: 100%
133M/133M [00:07<00:00, 45.7MB/s]
Downloading pytorch_model.bin: 100%
134M/134M [00:03<00:00, 44.1MB/s]
Downloading (…)nce_bert_config.json: 100%
57.0/57.0 [00:00<00:00, 1.18kB/s]
Downloading (…)cial_tokens_map.json: 100%
112/112 [00:00<00:00, 992B/s]
Downloading (…)da68f/tokenizer.json: 100%
466k/466k [00:00<00:00, 2.39MB/s]
Downloading (…)okenizer_config.json: 100%
362/362 [00:00<00:00, 9.37kB/s]
Downloading (…)662abda68f/vocab.txt: 100%
232k/232k [00:00<00:00, 1.77MB/s]
Downloading (…)abda68f/modules.json: 100%
387/387 [00:00<00:00, 8.57kB/s]
(3, 384) [[-0.03019609 -0.03029384 -0.09287093 ... -0.01719665  0.01654728
   0.10151423]
 [-0.02865791 -0.03035643 -0.12332754 ... -0.01281117  0.01556634
   0.08893888]
 [ 0.07090583 -0.05870947 -0.06741289 ... -0.01381944 -0.00529657
   0.07468784]]

Perhaps there is an implicit dependency on CUDA somewhere that's not checked conditionally based on what the specified compute flag indicates?

Thanks for your time and attention.

[michaelfeil]

@jcntrl Thanks for the detailed description.

The error should be indeed raised when importing sentence_transformers

Can you try again with?

pip install sentence-transformers torch

[jcntrl]

Interesting, I must have had a package conflict on my end; as in a new notebook with only:

!pip install sentence-transformers torch hf-hub-ctranslate2

and

from hf_hub_ctranslate2 import CT2SentenceTransformer
model_name_pytorch = "intfloat/e5-small"
model = CT2SentenceTransformer(
    model_name_pytorch, compute_type="int8", device="cpu", 
)
embeddings = model.encode(
    ["I like soccer", "I like tennis", "The eiffel tower is in Paris"],
    batch_size=32,
    convert_to_numpy=True,
    normalize_embeddings=True,
)
print(embeddings.shape, embeddings)
scores = (embeddings @ embeddings.T) * 100

it works, and results in an apparently correct:

(3, 384) [[-0.03019609 -0.03029384 -0.09287093 ... -0.01719665  0.01654728
   0.10151423]
 [-0.02865791 -0.03035643 -0.12332754 ... -0.01281117  0.01556634
   0.08893888]
 [ 0.07090583 -0.05870947 -0.06741289 ... -0.01381944 -0.00529657
   0.07468784]]

Thanks, I'll close the issue as it seems it's not one.