-
Something I don't yet understand in GPTQ is the significance of the dataset used for quantisation?
In Qwopqwop's GPTQ-for-LLaMa, the examples use `c4`. I've also seen him use `wikitext2` and `ptb`.…
-
The sample code uses random values to demonstrate functionality.
I'm unable to infer which tokenizer or module to use when I'm passing my own image via URL, local path, etc.
Same for defining si…
-
It's a nitpick, but I think Pythia uses Post-LayerNorm. However, in Lit-GPT, we may use Pre-LayerNorm:
https://github.com/Lightning-AI/lit-gpt/blob/0f021f3ad8cd8d6fe30b0ef721a7a0e2dac15898/lit_gpt/…
rasbt updated
6 months ago
-
### System Info
- System Info: 4xA6000 Workstation
- Full command line causing issues: `docker run --rm --entrypoint /bin/bash -itd --name "traclm-v1-3b-instruct" -v "path/to/folder":/data --gpus '"…
-
I'm building a custom architecture involving multiple existing architectures as sub-components (Pythia, RoBERTa, T5, etc).
Does this library support custom architectures? If not, could someone give…
-
Hello, I am trying to convert a model that is not on the itrex support list by referring to that [guide](https://github.com/intel/intel-extension-for-transformers/blob/main/intel_extension_for_transfo…
-
- [announcement][0]
- [base model, 3B][1]
- [instruct model, 3B][3]
- [chat model, 3B][2]
Hopefully this can be blazingly fast!
[0]: https://www.together.xyz/blog/redpajama-models-v1
[1]: ht…
-
I'm experiencing two issues below.
If there is any other way to improve the Inference speed while minimizing the loss of accuracy, please.
1. If use_quant is specified as false and executed in llm…
-
Falcon and gptneox seems to be supported in llama.cpp with a separate example - would be really cool to merge it in the binding and make it transparent for the user (to either load falcon, or gptneox …
-
Hi all,
is it supported with bs >1? found the following:
if (batch_size > 1)
MODEL_ASSERT(
("llama arch only supports continuous batching inference when giving multi prompts.",…