-
This is a nice tool, but I believe the exllama figures are incorrect -- there is an issue with the benchmarking, which, when corrected, shows that exllama is almost twice as fast as ctranslate2 with t…
-
I'm using a google collab that I use frequently but now it is throwing this problem. Any solution?
`
Traceback (most recent call last):
File "/usr/local/bin/whisperx", line 8, in
sys.exi…
-
Hi,
I am trying to convert codellama-34 to int8_bfloat16 using the following command:
`ct2-transformers-converter --model codellama/CodeLlama-34b-hf --copy_files tokenizer.model --output_dir…
-
Hi there,
When trying to do the model conversion via Ctranslate2 API (as same setup following as CLI), the conversion failed and popped the error 'File t does not exist in model openai/whisper-larg…
-
This might not be totally related to your project. But as someone who wrote multithreading library, I want to get your idea. Maybe this might give you idea to introduce new idea or improve the project…
-
I made a wrapper around CTranslate2, an API server. It batches requests, supports running these batches on multiple gpus (sort of round-robin), requests can have different priority levels, requests ar…
-
Hi I am new to ctranslate.
I am writing a fastapi server for llm inference with this code as a [base](https://github.com/OpenNMT/OpenNMT-py/pull/2489).
I want to write a input pydantic model to vali…
-
As the title suggest, if I try to set lib='basic' I get the following error:
`/home/ubuntu/.local/lib/python3.10/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object…
-
I've used the below code to convert and generate the output:
```
model_hf = "google/flan-t5-xl"
model_fast = "ct"
!ct2-transformers-converter --model $model_hf --output_dir $model_fast --force…
-
Hi,
Wasn't able to find great information on this, but do you know if https://github.com/tensorflow/tensor2tensor#translation supports exporting to Ctranslate2 format?
Thanks!
--Scot