I used A100(80G) to run the evaluate_zh.py script for evaluating baichuan model, but it occupied abundant GPU memory up to overflow. Then I found the model loaded without eval mode, meanwhile, it inferred without no_grad.
Reproducible example code
The Python snippets:
[https://github.com/baichuan-inc/Baichuan-7B/blob/6f3ef4633a90c2d8a3e0763d0dec1b8dc11588f5/evaluation/evaluate_zh.py#L97C13-L97C13](url)
self.model = model.eval()
https://github.com/baichuan-inc/Baichuan-7B/blob/6f3ef4633a90c2d8a3e0763d0dec1b8dc11588f5/evaluation/evaluate_zh.py#L103
Add on this line:
@torch.inference_mode()
Command lines:
Extra dependencies:
Steps to reproduce:
1.
2.
3.
Traceback
No response
Expected behavior
No response
Additional context
No response
Checklist
[X] I have provided all relevant and necessary information above.
[X] I have chosen a suitable title for this issue.
Required prerequisites
System information
conda environment torch=2.0.1 transformers=4.29.2 ...
Problem description
I used A100(80G) to run the evaluate_zh.py script for evaluating baichuan model, but it occupied abundant GPU memory up to overflow. Then I found the model loaded without eval mode, meanwhile, it inferred without no_grad.
Reproducible example code
The Python snippets:
Command lines:
Extra dependencies:
Steps to reproduce:
1. 2. 3.
Traceback
No response
Expected behavior
No response
Additional context
No response
Checklist