-
This project got my interest and wanted to give it a shot. Was attempting the getting started docker example and ran into issues:
**LocalAI version:**
Latest image
**Environment, CPU architectu…
-
Currently we are trying to run inference with pretrained BLOOM model. However, the loading takes very long due to DeepSpeed sharding in runtime. Since there is a pre-sharded version of BLOOM:
```
…
-
### System Info
- CPU architecture: x86_64
- CPU/Host memory size: 126G
- GPU properties
- GPU name: L4
- GPU memory size: 24GB
- Libraries
- TensorRT-LLM branch or tag (e.g., main, v0.…
-
I converted Astrid 1b CPU (https://huggingface.co/PAIXAI/Astrid-1B-CPU) to gguf and quantized it. Then i tried to run it using "main -m 1B/ggml-model-q4_1.gguf -n 128" and got this error:
error loa…
-
Hi, I wanted to convert the bin files of https://huggingface.co/stabilityai/japanese-stablelm-instruct-alpha-7b into .gguf, but it doesn't seem to work.
Based on the past issue(https://github.com/gge…
-
Hi, thanks for great work!
I want to use your code to build a `PipelineModule` object from LLama2. Here is my code:
```python
def load_model(neox_args):
config = transformers.AutoConfig.…
-
I am trying to save the Neuron model and deploy it to SageMaker as an endpoint. I noticed in the documentation, under [serialization support](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/li…
-
### System Info
- `transformers` version: 4.40.1
- Platform: Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.13
- Huggingface_hub version: 0.22.2
- Safetensors ver…
-
Hello everyone,
My question is regarding using IA3 with decoder-only LLMs containing "query_key_value" parameters, such as `GPTNeoX` and Falcon families (which I'll refer to as "query_key_value LLM…
-
Something I don't yet understand in GPTQ is the significance of the dataset used for quantisation?
In Qwopqwop's GPTQ-for-LLaMa, the examples use `c4`. I've also seen him use `wikitext2` and `ptb`.…