search

simon-ging / coot-videotext

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
Apache License 2.0
288 stars 55 forks source link

visualization of results #35

Closed liujiaqiKiKi closed 3 years ago

liujiaqiKiKi commented 3 years ago

Hi,I ran the project after the readme,but the results in Table 6 and Table 13 didn't appear.Would you please tell me what I should do? How do I visualize the results of the video text retrieval task and the video caption task?

simon-ging commented 3 years ago

Hi, the training and inference will generate translations as json, there you can find the captioning output. Search your repository for "*translation*.json" to find them.

We don't provide code to visualize videos. However since all videos where originally from youtube you can try to download them and visualize them yourself.

liujiaqiKiKi commented 3 years ago

Thank you for your reply.And how does the embedding of this project apply to video text retrieval?

simon-ging commented 3 years ago

I don't understand the question, please explain what you want to do.

With the embeddings you can 1) given a video sort all text embeddings by distance to this one video, and therefore do retrieval 2) same thing but from text to video 3) if you want to visualize the embeddings I suggest using t-SNE to reduce dimensionality to 2, then you can plot the 2d data with a scatter plot

liujiaqiKiKi commented 3 years ago

I'm a little confused. Once I have video and text embeddings,how should I use them for the video text retrieval task and the video captioning task? Is there another way to handle embedding?

simon-ging commented 3 years ago

There's a script to evaluate embeddings for retrieval: In Readme under "Extract your own embeddings" look for "compute retrieval on validation and train embeddings"

The Video captioning task is always based on embeddings so if you run the youcook2 captioning experiments, embeddings will be loaded from "provided_embeddings" folder

With a little digging in the code you should be able to work with the embeddings given the code in this repo.

liujiaqiKiKi commented 3 years ago

Hi,Thannk you for your reply! What should I do to get the table 6 and 14 in your paper?

simon-ging commented 3 years ago

Hi, I am very sorry but due to time limitations we cannot provide code or in-depth support for the data visualization used in our paper.