CeeZh
commented
2 months ago Hi,
Sorry for the late reply. I was quite overwhelmed these days.
For your question:
-
You say you use nucleus sampling with top_p=0.95 and choose k=5 for having 5 candidates. What temperature do you use? We used temperature=0.7. Basically, we follow the default haprams setting of LaViLa, except that we change caption-num-return-sequences to 5 to accelerate. [image: image.png]
-
Moreover, in the paper you say that you take the narration with the largest confidence score as the final caption. I assume you mean by that taking the candidate with the lowest perplexity (since LaViLa caption model returns output token ids together with the perplexity values)? Thanks for pointing it out! I double checked our code for captioning, and found that we were actually picking up the caption that has the largest perplexity. While we made fair comparisons in our paper, this is definitely a suboptimal solution.
-
Besides, I saw you reporting in the paper that you use a temperature=0.0 for the LLMs, but I see in the readme you provide example commands for the summarization task with temperature=1.0. So do you use temperature=1.0 for LLM in summarization task and temperature=0.0 in QA task? Yes, we only use temperature=1.0 in the summarization task because we found it better practically. For QA tasks we use temperature = 0.0.
Thanks for your questions, and good luck with your future research!
On Wed, Apr 24, 2024 at 3:50 AM maximotus @.***> wrote:
Hi ;)
for comparability reasons, it would be beneficial for the community to have insights into the full hyper-parameter setups. I am especially interested in the LaViLa captioning config to use with your provided fair checkpoint for EgoSchema. In detail, I need the following information:
- You say you use nucleus sampling with top_p=0.95 and choose k=5 for having 5 candidates. What temperature do you use?
- Moreover, in the paper you say that you take the narration with the largest confidence score as the final caption. I assume you mean by that taking the candidate with the lowest perplexity (since LaViLa caption model returns output token ids together with the perplexity values)?
- Besides, I saw you reporting in the paper that you use a temperature=0.0 for the LLMs, but I see in the readme you provide example commands for the summarization task with temperature=1.0. So do you use temperature=1.0 for LLM in summarization task and temperature=0.0 in QA task?
Clearification would be much appreciated! :)
Cheers, Maximotus
— Reply to this email directly, view it on GitHub https://github.com/CeeZh/LLoVi/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKXZD56IABN5UXGWXKRJQHLY65P35AVCNFSM6AAAAABGWMFHOCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DANRSGQ2TGMI . You are receiving this because you are subscribed to this thread.Message ID: @.***>
maximotus
Hi ;)
for comparability reasons, it would be beneficial for the community to have insights into the full hyper-parameter setups. I am especially interested in the LaViLa captioning config to use with your provided fair checkpoint for EgoSchema. In detail, I need the following information:
top_p=0.95and choosek=5for having 5 candidates. What temperature do you use?temperature=0.0for the LLMs, but I see in the readme you provide example commands for the summarization task withtemperature=1.0. So do you usetemperature=1.0for LLM in summarization task andtemperature=0.0in QA task?Clearification would be much appreciated! :)
Cheers, Maximotus