search

TIGER-AI-Lab / MEGA-Bench

This repo contains the code and data for "MEGA-Bench Scaling Multimodal Evaluation to over 500 Real-World Tasks"
https://tiger-ai-lab.github.io/MEGA-Bench/
Apache License 2.0
43 stars 1 forks source link

Could i just run image task without video task? #3

Closed Violettttee closed 3 weeks ago

Violettttee commented 3 weeks ago

our model evaluation repo doesnt finish video inference yet, and we just want to evaluate our model on images task.Is there any additional configuration could help this?(I see the input process has dealt with video or image file.If i just add additional process on input process, could evaluation goes properly?)

woodfrog commented 3 weeks ago

Good point. I want to confirm if you want to only evaluate your model with single-image tasks or if you want to exclude video tasks? If it's the former, you can consider using the singe-image setting.

Our data preprocessing pipeline extracts frames from a video file automatically based on pre-defined per-model frame sampling parameters. If your model can take multiple images, it can run with the video tasks in our benchmark.

Violettttee commented 3 weeks ago

Good point. I want to confirm if you want to only evaluate your model with single-image tasks or if you want to exclude video tasks? If it's the former, you can consider using the singe-image setting.

Our data preprocessing pipeline extracts frames from a video file automatically based on pre-defined per-model frame sampling parameters. If your model can take multiple images, it can run with the video tasks in our benchmark.

Thanks for your replying~ Does that mean the "single-image" subtasks have already exclude video file?I thought it means just use one image since the "query_media" filed is a list which may contain more than one image at first. I think we just want to exclude video tasks(our model can process multiple images but we just want to evaluate images task for now~).

woodfrog commented 3 weeks ago

Good point. I want to confirm if you want to only evaluate your model with single-image tasks or if you want to exclude video tasks? If it's the former, you can consider using the singe-image setting. Our data preprocessing pipeline extracts frames from a video file automatically based on pre-defined per-model frame sampling parameters. If your model can take multiple images, it can run with the video tasks in our benchmark.

Thanks for your replying~ Does that mean the "single-image" subtasks have already exclude video file?I thought it means just use one image since the "query_media" filed is a list which may contain more than one image at first. I think we just want to exclude video tasks(our model can process multiple images but we just want to evaluate images task for now~).

The single-image subset excludes all tasks that contain multiple images or videos. Using the Core set as an example, the single-image subset contains 273 tasks out of the full 440 Core tasks. We also exclude the image input in the 1-shot demonstration example so that the entire query only contains a single image. Maybe you can try the single-image subset first?

I see. We can indeed add a flag to enable only running image tasks. Some multi-image tasks contain a list of images extracted manually from a video by the task annotator, so I guess you don't want to evaluate your model on those tasks as well?

Violettttee commented 3 weeks ago

Oh,i got the point!I have been taking the "video" task as a real video task, which is different from images tasks.If its based on pre-defined per-model frame sampling parameters,i think it just consist of multiple images and we dont need to add "video" inputs and can just follow your process! Thanks a lot for your help~