Bad test results

[liziming5353]

I download llama1 and use apply_delta.py to generate ckpt. Then I use the original run_valley.py without any modifications to test on a test dataset, but get terrible results. May I ask if there is anything special that needs to be set up? Like prompt or something?

[RupertLuo]

Make sure the “<video>” token in your query. Can I see the results you got?

------------------ 原始邮件 ------------------ 发件人: "RupertLuo/Valley" @.>; 发送时间: 2023年11月13日(星期一) 下午2:55 @.>; @.***>; 主题: [RupertLuo/Valley] Bad test results (Issue #23)

I download llama1 and use apply_delta.py to generate ckpt. Then I use the original run_valley.py without any modifications to test on a test dataset, but get terrible results. May I ask if there is anything special that needs to be set up? Like prompt or something?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[RupertLuo]

Try this query, 'Describe this video concisely\<video\>.\n'

------------------ 原始邮件 ------------------ 发件人: "RupertLuo/Valley" @.>; 发送时间: 2023年11月13日(星期一) 下午3:11 @.>; @.**@.>; 主题: Re: [RupertLuo/Valley] Bad test results (Issue #23)

Make sure the “<video>” token in your query. Can I see the results you got? … ------------------ 原始邮件 ------------------ 发件人: "RupertLuo/Valley" @.>; 发送时间: 2023年11月13日(星期一) 下午2:55 @.>; @.>; 主题: [RupertLuo/Valley] Bad test results (Issue #23) I download llama1 and use apply_delta.py to generate ckpt. Then I use the original run_valley.py without any modifications to test on a test dataset, but get terrible results. May I ask if there is anything special that needs to be set up? Like prompt or something? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.>

query is "Describe this video concisely.\n

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[liziming5353]

The query is:

The response is:

I also print the outputs before self.process_response, and the result is:

No matter which video I input, it's always this result.

[RupertLuo]

Are your transformers version 4.28.0.dev?

[pratikkayal]

Hi I'm getting the same issue, I do have transformers version 4.28.0.dev Getting the following output: [',,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,....,,,,,,,,,,,,,,,,,,,,,,,,....................................................................................................,.,.........,.,,,,,,.,................................................................,,,,,,,.,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,................................................................................,.,.,...............................................................................................................................................................................,.,.,.,.....,.,.,.,.,................................................................................................................................................................................................,.,....,.,.,.,............................................................ (G..................................................................................-------']