search

YuanGongND / ltu

Code, Dataset, and Pretrained Models for Audio and Speech Large Language Model "Listen, Think, and Understand".
337 stars 27 forks source link

no evaluation script for open-set problem #11

Open alexaway opened 6 months ago

alexaway commented 6 months ago

Hello, thank you for providing a good idea of research on audio question answering. When I was testing, I found that there was no evaluation script for open-set problem in the code library, could you please provide one for reference?

YuanGongND commented 6 months ago

hi there,

Thanks for the question. The batch inference script does the inference for open-set questions:

https://github.com/YuanGongND/ltu/blob/0fa0923f9c9d04346486a28477ba69b7d957130c/src/ltu/inference_batch.py#L150-L153

The actual open-ended test set can be downloaded from https://www.dropbox.com/scl/fo/juh1dk9ltvhghuj0l1sag/h?rlkey=0n2cd5kebzh8slwanjzrfn7q6&dl=0, check open_as_gpt4.json and open_as_gpt3.json.

For your own question, you can use https://github.com/YuanGongND/ltu/blob/main/src/ltu/inference_gradio.py.

-Yuan