Closed lhtpluto closed 5 months ago
Hey @lhtpluto! Great to see you back here, diving into the depths of tech with us again. How's the journey been since our last encounter?
Based on the information you've provided, it seems that the uneven GPU usage might be due to how the workload is distributed across your GPUs. Here are some steps you can take to address this issue:
Ensure Equal Workload Distribution: Langchain-Chatchat allows you to specify the number of GPUs and their IDs, but it doesn't explicitly show how the GPUs are managed during the inference process. You should ensure that your application is correctly utilizing both GPUs. This can be achieved by implementing or ensuring there is logic within launch_all
or any relevant part of the code that handles GPU workload distribution. This might involve splitting the inference tasks equally and assigning them to each GPU.
Utilize PyTorch or TensorFlow's Built-in Functions: If Langchain-Chatchat is built on top of PyTorch or TensorFlow, make use of their built-in functions for multi-GPU support. For PyTorch, torch.nn.DataParallel
or torch.nn.parallel.DistributedDataParallel
can be used. For TensorFlow, tf.distribute.Strategy
can be utilized.
Explicit GPU Memory Management: The --max-gpu-memory
argument suggests there is a mechanism to limit GPU memory usage. Ensure this is correctly implemented to prevent one GPU from being overloaded. You might need to adjust the memory allocation based on the tasks' requirements.
Review and Optimize the Model Splitting (if applicable): If the inference model is split across GPUs, review the splitting logic. Uneven model splitting can lead to one GPU doing more work than the other.
Monitor and Debug GPU Utilization: Utilize tools like NVIDIA's nvidia-smi
to monitor GPU utilization and memory usage in real-time. This can help identify if the load balancing adjustments are effective.
Update to Latest Software Versions: Ensure that Langchain-Chatchat, along with its dependencies and the underlying deep learning frameworks (PyTorch, TensorFlow, etc.), are updated to their latest versions. Sometimes, performance improvements and bug fixes in newer versions can resolve such issues.
Without specific details on how the GPUs are being utilized within the launch_all
function or elsewhere in the code related to the inference process, these are general recommendations. For a more precise solution, reviewing the implementation details of GPU utilization within the Langchain-Chatchat inference process is necessary.
For more information on how Langchain-Chatchat handles GPU workload distribution, you can refer to the server/llm_api_stale.py file in the repository. This file contains the code that allows for flexible GPU workload distribution by specifying which GPUs to use, limiting the number of GPUs, managing GPU memory usage, and controlling model concurrency. This is achieved through a combination of command-line arguments and the setting of environment variables.
I hope this helps! If you have any more questions or need further clarification, feel free to ask.
因为是用fscaht加载,transformers的多卡是这样的
在fschat的model_adapter.py的def load_model这里进行修改。你再试试,我改好的文件被我升级fschat覆盖了,我好像没做备份,只能再去调试了哈哈。
问题描述 / Problem Description 用简洁明了的语言描述这个问题 / Describe the problem in a clear and concise manner. [BUG]显存使用率不平均(显存使用率40%和60%),推理过程中GPU负载也不平均(分别为35%和59%)
尝试通过device map映射方式,通过设置device_map参数(例如"auto", "balanced", "balanced_low_0", "sequential")改善,但找不到更改的位置
两张4090 ubuntu2204 Driver Version: 550.54.14 CUDA Version: 12.4 transformers 4.39.3 torch 2.1.2 torchaudio 2.1.2 torchvision 0.16.2 autoawq 0.2.3+cu121 /home/test/Langchain-Chatchat/AutoAWQ autoawq-kernels 0.0.6+cu121 /home/test/Langchain-Chatchat/AutoAWQ_kernels-main
环境信息 / Environment Information
当前使用的分词器:ChineseRecursiveTextSplitter 当前启动的LLM模型:['Qwen1.5-32B-Chat-AWQ'] @ cuda {'device': 'cuda', 'host': '0.0.0.0', 'infer_turbo': False, 'model_path': 'Qwen/Qwen1.5-32B-Chat-AWQ', 'model_path_exists': True, 'port': 20002} 当前Embbedings模型: bce-embedding-base_v1 @ cuda