dosubot[bot]
commented
5 months ago Hey @lhtpluto! Great to see you back here, diving into the depths of tech with us again. How's the journey been since our last encounter?
Based on the information you've provided, it seems that the uneven GPU usage might be due to how the workload is distributed across your GPUs. Here are some steps you can take to address this issue:
-
Ensure Equal Workload Distribution: Langchain-Chatchat allows you to specify the number of GPUs and their IDs, but it doesn't explicitly show how the GPUs are managed during the inference process. You should ensure that your application is correctly utilizing both GPUs. This can be achieved by implementing or ensuring there is logic within
launch_allor any relevant part of the code that handles GPU workload distribution. This might involve splitting the inference tasks equally and assigning them to each GPU. -
Utilize PyTorch or TensorFlow's Built-in Functions: If Langchain-Chatchat is built on top of PyTorch or TensorFlow, make use of their built-in functions for multi-GPU support. For PyTorch,
torch.nn.DataParallelortorch.nn.parallel.DistributedDataParallelcan be used. For TensorFlow,tf.distribute.Strategycan be utilized. -
Explicit GPU Memory Management: The
--max-gpu-memoryargument suggests there is a mechanism to limit GPU memory usage. Ensure this is correctly implemented to prevent one GPU from being overloaded. You might need to adjust the memory allocation based on the tasks' requirements. -
Review and Optimize the Model Splitting (if applicable): If the inference model is split across GPUs, review the splitting logic. Uneven model splitting can lead to one GPU doing more work than the other.
-
Monitor and Debug GPU Utilization: Utilize tools like NVIDIA's
nvidia-smito monitor GPU utilization and memory usage in real-time. This can help identify if the load balancing adjustments are effective. -
Update to Latest Software Versions: Ensure that Langchain-Chatchat, along with its dependencies and the underlying deep learning frameworks (PyTorch, TensorFlow, etc.), are updated to their latest versions. Sometimes, performance improvements and bug fixes in newer versions can resolve such issues.
Without specific details on how the GPUs are being utilized within the launch_all function or elsewhere in the code related to the inference process, these are general recommendations. For a more precise solution, reviewing the implementation details of GPU utilization within the Langchain-Chatchat inference process is necessary.
For more information on how Langchain-Chatchat handles GPU workload distribution, you can refer to the server/llm_api_stale.py file in the repository. This file contains the code that allows for flexible GPU workload distribution by specifying which GPUs to use, limiting the number of GPUs, managing GPU memory usage, and controlling model concurrency. This is achieved through a combination of command-line arguments and the setting of environment variables.
I hope this helps! If you have any more questions or need further clarification, feel free to ask.
zRzRzRzRzRzRzR
ilovesouthpark
问题描述 / Problem Description 用简洁明了的语言描述这个问题 / Describe the problem in a clear and concise manner. [BUG]显存使用率不平均（显存使用率40%和60%），推理过程中GPU负载也不平均（分别为35%和59%）
尝试通过device map映射方式，通过设置device_map参数（例如"auto", "balanced", "balanced_low_0", "sequential"）改善，但找不到更改的位置
两张4090 ubuntu2204 Driver Version: 550.54.14 CUDA Version: 12.4 transformers 4.39.3 torch 2.1.2 torchaudio 2.1.2 torchvision 0.16.2 autoawq 0.2.3+cu121 /home/test/Langchain-Chatchat/AutoAWQ autoawq-kernels 0.0.6+cu121 /home/test/Langchain-Chatchat/AutoAWQ_kernels-main
环境信息 / Environment Information
当前使用的分词器：ChineseRecursiveTextSplitter 当前启动的LLM模型：['Qwen1.5-32B-Chat-AWQ'] @ cuda {'device': 'cuda', 'host': '0.0.0.0', 'infer_turbo': False, 'model_path': 'Qwen/Qwen1.5-32B-Chat-AWQ', 'model_path_exists': True, 'port': 20002} 当前Embbedings模型： bce-embedding-base_v1 @ cuda