search

line / lighthouse

A user-friendly library for reproducible video moment retrieval and highlight detection.
https://www.arxiv.org/abs/2408.02901
Apache License 2.0
70 stars 7 forks source link

Is there any requirement for the input video? #2

Closed whycantfindaname closed 1 month ago

whycantfindaname commented 1 month ago

I run the api_example/demo.py using cgdetr on a video of length 104s and it report the bug as 

Traceback (most recent call last):
  File "/home/桌面/video_highlight/lighthouse/api_example/demo.py", line 29, in <module>
    model.encode_video('/home/liaowenjie/桌面/video_highlight/video/video1_part_1.mp4')
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/lighthouse/models.py", line 86, in encode_video
    video_feats = self.feature_extractor.encode_video(video_path)
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/lighthouse/feature_extractor.py", line 192, in encode_video
    slowfast_frames = self.sf_loader.read_video_from_file(video_path)
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/lighthouse/video_loader.py", line 223, in read_video_from_file
    video = self.preprocess(video, info)
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/lighthouse/slowfast_preprocess.py", line 183, in __call__
    tensor = self._pad_clips(tensor, self.padding_mode)
  File "/home/anaconda3/envs/lighthouse/lib/python3.9/site-packages/lighthouse/slowfast_preprocess.py", line 174, in _pad_clips
    z = th.cat(n * [tensor[-1:, :, :, :, :]])

And I test on the other video of length 150s, it succeed but it gives a negative pred_relevant_window [-0.1586, 5.4191, 0.9571]. I notice the original code have the limitation that the video length should be under 150s. So is this the problem of the video or the problem of the cgdetr?

awkrail commented 1 month ago

Thank you for your test.

I run the api_example/demo.py using cgdetr on a video of length 104s and it report the bug as

This is a bug. I will fix this.

And I test on the other video of length 150s, it succeed but it gives a negative pred_relevant_window [-0.1586, 5.4191, 0.9571].

You need to clip max(0, start) / min(video_length, end). This is post-processing, but is not implemented now.

I notice the original code have the limitation that the video length should be under 150s. So is this the problem of the video or the problem of the cgdetr?

The max video length is 150s due to the QVHighlights limitation.

awkrail commented 1 month ago

@whycantfindaname I reproduced the error. But please wait for 1 week due to the conference deadline. Thanks.

whycantfindaname commented 1 month ago

@whycantfindaname I reproduced the error. But please wait for 1 week due to the conference deadline. Thanks.

It's ok. Thanks for your contribution!

awkrail commented 1 month ago

@whycantfindaname Hi, I fixed the issue. Could you try it again? If you have any problems, please re-open the issue.