Why the result of temporal video grounding is always the multiple of 5?

[zhengrongz]

Hi! When I am trying my own videos in the demo.ipynb, the results of temporal video grounding are always the multiple of 5. e.g. 0-5 seconds, 10-15 seconds, 0-10 seconds. I can't get the exact result to the decimal point as you show. I don't change any configs in the demo.ipynb except the video_path and the prompt. The prompt I use is the "Localize the visual content described by the given textual query 'person picks up a bag' in the video, and output the start and end timestamps in seconds." the total length of my videos are average 20-30s.

[RenShuhuai-Andy]

Hi, can you upload your videos? You can do so by clicking the buttom of attach files

[zhengrongz]

https://github.com/RenShuhuai-Andy/TimeChat/assets/63049360/13c2cf0e-39fd-44c8-a074-bfe9f7bbbbb2

ok! that's one of my videos, and I want to know the happening time of some actions. e.g. "person picks up a bag", "person sweep the floor" in this video.

[RenShuhuai-Andy]

I have reviewed your case, and indeed, the output is typically 0-5 seconds.

I also examined the predictions on the Charades dataset, where some predicted timestamps are multiples of 5:

I speculate that this may be due to many queries in the training data corresponding to annotations like 0-5s, 5-10s, 10-15s, etc. However, I believe this does not affect the overall evaluation?

[zhengrongz]

ok, I got it! Thanks for your help! I thought at first that there was something wrong with my setup that was causing some weird errors in the model. One last question, do you just use the same prompt as I mentioned above in your test case for Charades dataset?

[RenShuhuai-Andy]

No, for the evaluation on Charades, I used the prompt from https://github.com/RenShuhuai-Andy/TimeChat/blob/master/prompts/tvg_description_zeroshot.txt.

The prompt you mentioned above is only used for your provided video.