04-GLM-4-9B-Chat vLLM 部署调用显存溢出问题

[SongHY-13]

1、在服务器上4张T4卡上尝试运行vllm_demo，首先是告诉我T4不支持bfloat精度，ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5. You can use float16 instead by explicitly setting thedtype flag in CLI, for example: --dtype=half.
2、前面3个步骤fastapi、streamlit、langchain跟着教程做都能运行，所以很奇怪， 3、然后换了精度为“float16”，这时候报错：orch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 214.00 MiB. GPU 0 has a total capacty of 14.58 GiB of which 127.56 MiB is free. Including non-PyTorch memory, this process has 14.45 GiB memory in use. Of the allocated memory 14.19 GiB is allocated by PyTorch, and 1.15 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 4、修改了max_model_len,max_tokens等都没用

[Bald0Wang]

按顺序跑，出问题的这一步配置在最后的测评脚本吗还是哪里没说清楚~ 如果在测评脚本是不是前面的vllm脚本没有关闭，还在占用。如果是一开始就满了看看是不是fastapi这些还挂着？

[SongHY-13]

可以跑通了，之前的都关闭了，是运行这个脚本出的问题： ， 应该是GPU适配的关系，我在 llm = LLM(model=model, tokenizer=tokenizer, max_model_len=max_model_len,trust_remote_code=True,dtype="float16",tensor_parallel_size =4,disable_custom_all_reduce = True) 主要是修改了精度和,tensor_parallel_size分成4份执行，就可以跑通了

[Bald0Wang]

参考issue 这里也有一个差不多的功能，可以参考这里。感谢提issue，其他小伙伴也能参考学习。

[13428116504]

把这个值调低一点就可以了，gpu_memory_utilization 0.60或者更小