errors in MME evaluation

[liuliAI]

Describe the issue

Issue: errors in MME evaluation Command:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh

Log:

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)`
Traceback (most recent call last):
  File "/data/LLaVA/playground/data/eval/MME/convert_answer_to_mme.py", line 43, in <module>
    GT = get_gt(
  File "/data/LLaVA/playground/data/eval/MME/convert_answer_to_mme.py", line 35, in get_gt
    question, answer = line.strip().split('\t')
ValueError: not enough values to unpack (expected 2, got 1)
=========== Perception ===========
Traceback (most recent call last):
  File "/data/LLaVA/playground/data/eval/MME/eval_tool/calculation.py", line 164, in <module>
    cal.process_result(results_dir)
  File "/data/LLaVA/playground/data/eval/MME/eval_tool/calculation.py", line 98, in process_result
    lines = open(task_txt, 'r').readlines()
FileNotFoundError: [Errno 2] No such file or directory: 'answers/llava-v1.5-13b/existence.txt'

[wijayarobert]

I also get the same error. Did you manage to resolve it?

[F-Yuan303]

I also get the same error. Did you manage to resolve it?

[zhenyu228]

I also get the same error. Did you manage to resolve it?

[zhenyu228]

Describe the issue

Issue: errors in MME evaluation Command:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh

Log:

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)`
Traceback (most recent call last):
  File "/data/LLaVA/playground/data/eval/MME/convert_answer_to_mme.py", line 43, in <module>
    GT = get_gt(
  File "/data/LLaVA/playground/data/eval/MME/convert_answer_to_mme.py", line 35, in get_gt
    question, answer = line.strip().split('\t')
ValueError: not enough values to unpack (expected 2, got 1)
=========== Perception ===========
Traceback (most recent call last):
  File "/data/LLaVA/playground/data/eval/MME/eval_tool/calculation.py", line 164, in <module>
    cal.process_result(results_dir)
  File "/data/LLaVA/playground/data/eval/MME/eval_tool/calculation.py", line 98, in process_result
    lines = open(task_txt, 'r').readlines()
FileNotFoundError: [Errno 2] No such file or directory: 'answers/llava-v1.5-13b/existence.txt'

Solution： Modify the get_gt() function in convert_answer_to_mme.py to the following code. This code adds debugging statements to the original function to identify where the error occurs, while also skipping these errors.

def get_gt(data_path):
    GT = {}
    for category in os.listdir(data_path):
        category_dir = os.path.join(data_path, category)
        if not os.path.isdir(category_dir):
            continue
        if os.path.exists(os.path.join(category_dir, 'images')):
            image_path = os.path.join(category_dir, 'images')
            qa_path = os.path.join(category_dir, 'questions_answers_YN')
        else:
            image_path = qa_path = category_dir
        assert os.path.isdir(image_path), image_path
        assert os.path.isdir(qa_path), qa_path
        for file in os.listdir(qa_path):
            if not file.endswith('.txt'):
                continue
            file_path = os.path.join(qa_path, file)
            with open(file_path, 'r') as f:
                for line_number, line in enumerate(f, start=1):
                    line = line.strip()
                    if not line:  # Skip empty lines
                        continue
                    parts = line.split('\t')
                    if len(parts) != 2:  # If the line format is incorrect, output the file name and line number
                        print(f"Skipping malformed line in file {file} at line {line_number}: {line}")
                        continue
                    try:
                        question, answer = parts
                        GT[(category, file, question)] = answer
                    except ValueError as e:
                        print(f"Error processing line {line_number} in file {file}: {line}")
                        print(f"Exception: {e}")
                        continue
    return GT

It is evident that the issue occurred in ./playground/data/eval/MME/eval_tool/readme.txt