-
triton nvcr.io/nvidia/tritonserver 23.12-py3
I have 4 RTX4090 Nvidia graphics cards, and my model is an ensemble model, which can be understood as preprocessing+inference. In config.pbtxt, the gpus…
-
### Issues Policy acknowledgement
- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)
### Where…
-
/kind feature
**Describe the solution you'd like**
While deploying transformer/predictor as separate services gives you the flexibility of running different replicas for transformer and predictor.…
-
Whereas /v1/chat/completions succeeds , the same body /v1/embeddings returns a 404 for a similar body
I was hoping to get the embedding output vector for an image that uses the openbmb/MiniCPM-V-2…
-
### Describe the bug
So my issue is this error, I was working with same 'HF_token' having the write permission and I am working with Mistral Nemo 12B Instruct , the model was working well from last…
-
I am using the InternVL2-40B-AWQ model and performing video inference according to the multi-images inference paradigm. Each video is sampled into 24 frames, and the prompt is as follows. My questions…
-
![image](https://github.com/user-attachments/assets/4fa65867-6cbf-489f-9b12-0ba881b1347e)
I have 2 model folders for llama3, one is the original and another is the finetuned, how to config to use the…
-
**System information**
- Android Device information: `samsung/dm3qxeea/dm3q:14/UP1A.231005.007/S918BXXS3BWK5:user/release-keys` (reproduced on any device I tested)
- Bundled TFLite version from [Mav…
Qheb updated
6 months ago
-
**Is your feature request related to a problem? Please describe.**
At present text generation is only supported for http_server.cc and not supported in sagemaker_server.cc. This was verified using vL…
-
I found that the entire code inference process does not use a quantizer to discretize the encoding representation, nor does it use `VectorQuantizerWithInputProjection` for conversion. It is directly h…