search

IAAR-Shanghai / CRUD_RAG

CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
https://arxiv.org/abs/2401.17043
233 stars 20 forks source link

How much GPU memory is required for default set running? any comments? #5

Closed yxm0513 closed 6 months ago

yxm0513 commented 6 months ago

1 GPU Memery issue (CUDA out of memory.)

Output saved at ./output/test_top2_Qwen_7B_Chat/ContinueWriting_qwen7b.json!

0%| | 0/5 [00:00<?, ?it/s]2024-04-03 14:51:50.480 | WARNING | src.llms.base:safe_request:43 - OutOfMemoryError('CUDA out of memory. Tried to allocate 104.00 MiB (GPU 1; 7.92 GiB total capacity; 7.74 GiB already allocated; 7.69 MiB free; 7.75 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')

2024-04-03 14:51:50.498 | WARNING | src.llms.base:safe_request:43 - OutOfMemoryError('CUDA out of memory. Tried to allocate 16.00 MiB (GPU 1; 7.92 GiB total capacity; 7.74 GiB already allocated; 7.69 MiB free; 7.75 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF')

2 swith to CPU, does ZeroDivisionError impacts final results?

100%|██████████| 5/5 [07:06<00:00, 85.39s/it] Output saved at ./output/test_top2_Qwen_7B_Chat/Summary_qwen7b.json! 100%|██████████| 5/5 [30:58<00:00, 371.78s/it] Output saved at ./output/test_top2_Qwen_7B_Chat/ContinueWriting_qwen7b.json! 0%| | 0/5 [00:00<?, ?it/s]2024-04-03 11:19:09.803 | WARNING | src.metric.common:wrapper:19 - ZeroDivisionError('float division by zero') 2024-04-03 11:19:09.803 | WARNING | evaluator:process_data_point:94 - TypeError('cannot unpack non-iterable NoneType object') 80%|████████ | 4/5 [1:04:08<10:57, 657.84s/it]2024-04-03 11:19:50.945 | WARNING | src.metric.common:wrapper:19 - ZeroDivisionError('float division by zero') 2024-04-03 11:19:50.946 | WARNING | evaluator:process_data_point:94 - TypeError('cannot unpack non-iterable NoneType object') 100%|██████████| 5/5 [1:04:09<00:00, 769.96s/it] Output saved at ./output/test_top2_Qwen_7B_Chat/HalluModified_qwen7b.json! 0%| | 0/5 [00:00<?, ?it/s]2024-04-03 12:07:17.097 | WARNING | src.metric.common:wrapper:19 - ZeroDivisionError('float division by zero') 2024-04-03 12:07:17.098 | WARNING | evaluator:process_data_point:94 - TypeError('cannot unpack non-iterable NoneType object') 100%|██████████| 5/5 [47:27<00:00, 569.47s/it] Output saved at ./output/test_top2_Qwen_7B_Chat/QuestAnswer1Doc_qwen7b.json! 0%| | 0/5 [00:00<?, ?it/s]2024-04-03 12:50:40.766 | WARNING | src.metric.common:wrapper:19 - ZeroDivisionError('float division by zero') 2024-04-03 12:50:40.767 | WARNING | evaluator:process_data_point:94 - TypeError('cannot unpack non-iterable NoneType object') 100%|██████████| 5/5 [43:21<00:00, 520.36s/it] Output saved at ./output/test_top2_Qwen_7B_Chat/QuestAnswer2Docs_qwen7b.json! 20%|██ | 1/5 [37:01<2:28:04, 2221.07s/it]2024-04-03 13:34:40.221 | WARNING | src.metric.common:wrapper:19 - ZeroDivisionError('float division by zero') 2024-04-03 13:34:40.222 | WARNING | evaluator:process_data_point:94 - TypeError('cannot unpack non-iterable NoneType object') 100%|██████████| 5/5 [43:52<00:00, 526.53s/it] Output saved at ./output/test_top2_Qwen_7B_Chat/QuestAnswer3Docs_qwen7b.json!

3 do you have results for models, like qwen, gpt? where to fetech it?

haruhi-sudo commented 6 months ago

  1. A GPU with more than 24GB of Cuda memory is required. If you feel that the cuda memory is not enough, you can try using frameworks like vllm, deepseed, etc. to run the Qwen model.

  2. I have not tested the model on a CPU, so I'm not sure if there would be any impact. I suggest carefully checking the output files and the generated content from the model to determine if the model has generated appropriate content.

  3. If you need experiment results, please contact the email address s1583050085@gmail.com.