What does the dimension of data mean?

[kxkaixin]

Thank you very much for the code you shared. Because my device cannot run this code, I would like to ask you what the data dimensions of the original text and video are and what they mean respectively.text_data = data['text'] and video_data = data['video'].

Text_features = self.clip.encode_text(text_data) and video_features = self.clip.encode_image(video_data) . What are the resulting data dimensions for Text_features and video_features, and what do they mean?

Looking forward to your reply.

 def forward(self, data, return_all_frames=False):
        batch_size = data['video'].shape[0]
        text_data = data['text']
        video_data = data['video']
        video_data = video_data.reshape(-1, 3, self.config.input_res, self.config.input_res) 
        if self.config.huggingface:
            text_features = self.clip.get_text_features(**text_data)
            video_features = self.clip.get_image_features(video_data)
        else:
            text_features = self.clip.encode_text(text_data)
            video_features = self.clip.encode_image(video_data)
        video_features = video_features.reshape(batch_size, self.config.num_frames, -1)#[16,2,512]
        video_features_pooled = self.pool_frames(text_features, video_features)

[NoelVouitsis]

Thank you for taking interest in our work!

The text data is the resulting output from the CLIP tokenizer and the video data is the sampled set of frames from the original video reshaped to a height and width of 224x224 in our code.

The dimensions of text_features and video_features are [number of texts, 512] and [number of videos, number of frames, 512] respectively, where 'number of texts' and 'number of videos' are both set to the batch size in our code. The features represent the embeddings of the text and the video frames in the CLIP latent space.

[kxkaixin]

Your reply has been received. Thank you very much for your reply. Thanks again!