-
... or whether we need to tweak some settings
GitHub: https://github.com/openlm-research/open_llama
HuggingFace: https://huggingface.co/openlm-research/open_llama_7b_preview_300bt
---
edit…
-
paddle_lite_opt版本v2.9
$ paddle_lite_opt --model_file qat_model.pdmodel --param_file qat_model.pdiparams --optimize_out_type=naive_buffer --optimize_out=trans_qat --valid_targets=arm --quant_model=t…
-
Hi, playing around with quantizing a 70b model. Even with 4x A100s 80gb each it is OOM'ing, is this normal? It seems to be splitting the model on the GPUs correctly but not sure if it's splitting infe…
-
I was looking at a past issue (see below) regarding how to set-up my custom dataset for INT8 calibration. I was just wondering what subset of the dataset indicated in the YAML file is used for calibra…
-
### 🐛 Describe the bug
Hi, when I try to fuse the operators (i.e., `Conv2d` and `BatchNorm2d`), the fusion seems to fail. Fused operators should be `ConvBatchNorm2d` but in fact only `Conv2d` is pr…
-
At least for me, the per-epoch/step saving function of transformers trainer only saves the intermediate adapter_model.bin, but this does not include the trainable embed and norm layers. Is there some …
-
Hi,
I reimplemented SQuant in my code.
The result for ResNet-18 (top1 acc: 71.47) w4a32 quantization is 69.734%, it works fine.
But when I combined it with MSE to test the orthogonality,
the res…
-
Hi, I am new to LLM quantization and wanted to use text-generation-webui to run a local LLM chatbot.
ExLlamaV2 worked great when I applied it to the 13B model with 8bpw, but when it came to the 70B…
-
Dear Author,
thank you so much for the project. I have read your results, and have a question regarding the implementation of the PTQ. Does it support INT4/INT6 or other?
-
## Expanding Directions for Steps Following EdgeTPU Import
### Here is our project introduction
Project Introduction : #1621
### Background
To fully utilize EdgeTPU during inference, the m…