-
## Motivation
In the current technological landscape, Generative AI (GenAI) workloads and models have gained widespread attention and popularity. Large Language Models (LLMs) have emerged as the dom…
-
It does not appear that the model artifacts needed to explore quantized models are made public, specifically with the python module.
For instance, with a non-quantized supervised model class, you c…
-
I'm encountering a problem where the 'render' button doesn't appear after loading prompts. I've tried reinstalling the software 3-4 times, even with a fresh installation, but the issue persists. The p…
-
The classes in the `gr.iti.mklab.visual.quantization` package train the various models used by the classes in the `gr.iti.mklab.visual.datastructures` package. It's hard to construct the data pipeline…
-
This issue is to provide some context and discussion for current quantized matmuls compared against what is done for [GGML in llama.cpp](https://github.com/ggerganov/llama.cpp/blob/master/ggml.c).
…
-
When using spherical=True during Kmeans clustering, the centroids are normalized after each iteration. The documentation can be found here: https://github.com/facebookresearch/faiss/wiki/Faiss-buildin…
-
Milvus supports on-disk index https://milvus.io/docs/disk_index.md. This should reduce compute usage drastically. DiskANN is enabled by default but I think we need to make sure the volume inside the …
-
-
As per mentioned in the paper ["Product quantization for nearest neighbor search"](https://hal.inria.fr/inria-00514462/document) and implementation using registers to store look-up tables are providin…
-
Hello, I want to deploy llama-3-8b quantized model using tritonserver I followed below steps to do this:
1. create container with nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 base image.
3.…