Closed IFICL closed 1 month ago
For this basic model, YES. This is actually an interesting problem, as the embedding of vision features significantly influences the generation regardless of their own capability. We reference the CLIP4Clip framework in our design (so it's labelled CLIP4CLIP), and when removing the Position Embedding and setting the patch-num to 1 per frame, it results in the current structure utilized in our basic model.
Thanks for the clarification.
Hi, I have some questions regarding the CLIP4CLIP encoder you are using. From what I saw from the
summary.jsonl
, thefea_encoder_name
is"openai/clip-vit-large-patch14"
which is clip embedding. Need clarifications for the video encoder. How do you encode the video features? Do you actually use CLIP to get embedding pre frame and then concat them?