-
### Your current environment
docker latest for 0.5.3
```
docker pull vllm/vllm-openai:latest
docker run -d --restart=always \
--runtime=nvidia \
--gpus '"device=1"' \
--shm-size…
-
Looking for onnx version of microsoft/Phi-3-vision-128k-instruct
Onnx files don't seem to be on Hugging Face.
Also, Does the onnxruntime-genai support multiple GPUs on the same PC?
Thanks,
Ash
AshD updated
5 months ago
-
### 🚀 The feature, motivation and pitch
InternVL2 is currently the most powerful open-source Multimodal Large Language Model (MLLM). The InternVL2 family includes models ranging from a 2B model, suit…
-
Please comment here any model requests for:
- Models to be added to https://huggingface.co/neuralmagic
- Example scripts within `llm-compressor`
-
### First Check
- [X] This is not a feature request.
- [X] I added a very descriptive title to this issue (title field is above this).
- [X] I used the GitHub search to find a similar issue and didn'…
-
Hi - thanks so much for making this repo!
I just ran the benchmark on my 32GB M1 Macbook Pro and I'm getting tps numbers roughly 60% of what was reported. Any idea on what might be going on?
Se…
-
Hey, I am trying to integrate your phi-3-vision script into LitServe.
How can I use the already loaded image (bytes) and pass it to the processor? (I don't want to save it as local/tmp file)
Tha…
-
I'm wondering what causes this error?
Do I have to set --version phi3 during pre-training stage? I use --version plain in pre-train stage and --version phi3 in fine-tune stage. Is this the correct s…
-
### Current Behaviour
# converts the data types of the columns in the DataFrame to more appropriate types,
# useful for improving the performance of calculations.
…
-
### Anything you want to discuss about vllm.
I was wondering why does this happen? I am new to this space and was playing around with different machines, models and frameworks.
I am able to infere…