what's the difference between get_context_emb() and get_context_emb_sim()?

TencentARC / ST-LLM

[ECCV 2024🔥] Official implementation of the paper "ST-LLM: Large Language Models Are Effective Temporal Learners"

Apache License 2.0

125 stars 4 forks source link

what's the difference between get_context_emb() and get_context_emb_sim()? #13

Closed dragen1860 closed 5 months ago

dragen1860 commented 6 months ago

Dear all: I noticed that you use d get_context_emb_sim() in demo.py but used get_context_emb() in demo_gradio.py. I wonder both the two function are embedding the image&text tokens, what is the difference? thank you.

farewellthree commented 5 months ago

Both demo and demo_gradio use get_context_emb_sim. get_context_emb_sim is suitable for InstructBLIP, while get_context_emb is suitable for MiniGPT-4.