-
Dear authors:
I am wondering if I could get some demos on config which is used to train a Language Model with ZeRO-Infinity?
It confused me a lot that how to config the "offload_param" and "offload_op…
-
I am using the latest vllm docker image, trying to run Mixtral 8x7b model quantized in AWQ format. I got error message as below:
```
INFO 12-24 09:22:55 llm_engine.py:73] Initializing an LLM engine …
-
-
See:
```
CREATE TABLE input_data (id UInt64, data String) ENGINE = MergeTree() ORDER BY id;
CREATE TABLE url_target (id UInt64, data String) ENGINE = URL('http://some.host.com/v1/route', JSON);
CR…
-
-
> I have test HFNet with TF-TRT on Tensorflow 1.x and Tensorflow 2
> When using TensorFlow 1.x, dynamics_op must set to true and it takes long time of initialization. When using Tensorf…
-
-
## Environment
- **GPUs**: 4x NVIDIA A100 (80GB) (nvlink. azure Standard_NC96ads_A100_v4)
- **TensorRT-LLM Version**: 0.15.0.dev2024102200
- **Environment**: Docker container
- **Memory Usage per GPU…
-
```
from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM, AutoConfig, get_scheduler
import deepspeed
model = AutoModelForCausalLM.from_pretrained("models/opt-6.7b")
tokenizer = …
-
I am facing this error while trying to fine-tune the phi3.5 vision model with lora.
created the virtual env based on the environment.yaml file,
all the library version are as mentioned.
```
[r…