hanoonaR
commented
11 months ago Hi @Kratos-Wen,
Thank you for your interest in our work. 1) I would suggest ignoring them, as they are warnings. (https://github.com/mbzuai-oryx/Video-ChatGPT/issues/28#issuecomment-1651526374)
2) No, you are correct. Apologies for not making this clear in the released code. Here in the case of benchmarking, we evaluate five criteria: correctness, detained orientation, context, temporal, and consistency. Except for consistency, every other criterion requires only one question per sample. However in the case of consistency, what we intend to do is when asked about a similar concept in two different perspectives, how does the model respond So as you see here in the evaluation, we make use of both questions.
Hope its clear.
Kratos-Wen
Hi, thank you for the great work!
I've encountered a couple of issues while running inference following the instructions in "QuantitativeEvaluation.md":
H264 decoding error: When I attempt to run the inference, the process doesn't terminate, and I repeatedly receive the following error: [h264 @ 0x35aaf840] mmco: unref short failure
Question retrieval in 'run_inference_benchmark_general.py': I noticed that the script expects to find the question using a 'Q' key in the JSON file. However, in "consistency_qa.json", the questions are keyed with "Q" followed by a number (e.g., "Q1", "Q2", etc.). This discrepancy seems to prevent the correct retrieval of the questions.
Is this a known issue, or am I possibly misinterpreting the correct usage? Any guidance on these issues would be greatly appreciated. Thank you in advance for your assistance!