-
I am referring to https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_distributed.py example. The suggestion in the example is not to set num_gpus when tensor parallel is used. H…
-
# Background research
## Readings
https://lilianweng.github.io/posts/2023-06-23-agent/
> * Finite context length: The restricted context capacity limits the inclusion of historical information, de…
-
### System Info
PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-1…
-
### Your current environment
```text
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
…
-
- [ ] Create philosophical shorts for why LLM may actually "understand"
- [ ] Create a weekly target
- [ ] Reflect on how I would trickle from year to daily vision
- [ ] Create gigs on fastwork
- [ ] …
-
### Reminder
- [X] I have read the README and searched the existing issues.
### Reproduction
```
USE_MODELSCOPE_HUB=1 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.run \
…
-
(songdh) [root@localhost server_llm]# python -m vllm.entrypoints.api_server --model $model_path --tokenizer $model_path --tensor-parallel-size $GPUS --dtype auto --port $port -…
-
> PPO 阶段不支持 DeepSpeed,仅支持 Accelerate
Accelerate 的启动命令应该是怎么样呢,可以指定几个gpu吗?
_Originally posted by @yuye2133 in https://github.com/hiyouga/LLaMA-Factory/issues/831#issuecomment-1709987…
-
### Question Validation
- [X] I have searched both the documentation and discord for an answer.
### Question
my gpu is 24G memory,so, its possiable to spread models to 2 GPUs。
torch.cuda.OutOf…
OKC13 updated
4 months ago
-
### Your current environment
**When I send the cl:**
```
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2-1.5B-Instruct --served-model-name Qwen2-7B-Instruct-lora --max-model-len=204…