Video can't generate problems

ShaoDu commented 5 years ago

when I run the command： python summary2video.py -p log/summe-split0/result.h5 -d video_frames/ -i 0 --fps 30 --save-dir log --save-name summary.mp4 this occured： OpenCV: FFMPEG: tag 0x5634504d/'MP4V' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)' OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v' Traceback (most recent call last): File "summary2video.py", line 43, in frm2video(args.frm_dir, summary, vid_writer) File "summary2video.py", line 27, in frm2video frm = cv2.resize(frm, (args.width, args.height)) cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4044: error: (-215:Assertion failed) !ssize.empty() in function 'resize' My generation is to use ffepeg operation video frame,One of my format is generated according to the requirements of the code format， why such a mistake？Can you help me?

SinDongHwan commented 5 years ago

on README, Visualize summary

You can use summary2video.py to transform the binary machine_summary to real summary video.

You need to have a directory containing video frames.

The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a .mp4 video.

maybe, it seems that there aren't frames on "video_frames" directory. but i don't know which frames yet.

Did you solve this problem?

ShaoDu commented 5 years ago

Thank you very much for your reply! I haven't solved this problem for the moment, I just tried to split the video to some frames and save them in a folder, but I don't know how the summary2video-py uses these frames to generate the MP4 video summary, and the result is always that the corresponding frame is not found.

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"notifications@github.com; 发送时间: 2019年1月29日(星期二) 下午4:39 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"pytorch-vsumm-reinforce@noreply.github.com; 抄送: "邵都"2577414131@qq.com; "Author"author@noreply.github.com; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15)

on README, Visualize summary

You can use summary2video.py to transform the binary machine_summary to real summary video.

You need to have a directory containing video frames.

The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a .mp4 video.

maybe, it seems that there aren't frames on "video_frames" directory. but i don't know which frames yet.

Did you solve this problem?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

SinDongHwan commented 5 years ago

Did you see this line in summary2video.py?

we have to formate filename as "000001.jpg, 000002.jpg, ..." and i found about number of video frames.

import h5py
h5_file_name = "datasets/eccv16_dataset_summe_google_pool5.h5"
f = h5py.File(h5_file_name,'r')
print(f['video_1']['n_frames'].value)

i can see "4494".

we can see more information about each h5 files on datasets/readme.txt .

i will try to find fps to convert frames from video correctly.

ShaoDu commented 5 years ago

Yes, I do. I made the format of video frame according to your format requirements. So do you mean save all video frames as.h5 instead of saving them in a folder and reading them? If that's the case, I'll try again.Because I tried in the folder to save.

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"notifications@github.com; 发送时间: 2019年1月29日(星期二) 下午5:38 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"pytorch-vsumm-reinforce@noreply.github.com; 抄送: "邵都"2577414131@qq.com; "Author"author@noreply.github.com; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15)

Did you see this line in summary2video.py?

we have to inform filename "000001.jpg, 000002.jpg, ..." and i found about number of video frames. import h5py h5_file_name = "datasets/eccv16_dataset_summe_google_pool5.h5" f = h5py.File(h5_file_name,'r') print(f['video_1']['n_frames'].value) i can see "4494". we can see more information about each h5 files on datasets/readme.txt . i will try to find fps to convert frames from video correctly.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

SinDongHwan commented 5 years ago

No, we should save frames in folder. I told you that read dataset/readme.txt to convert frames from video correctly. Parameter "summary" of frm2video function consists of [0,1,0,1,0,0,0,...]. this length of list( [0,1,0,1,0,0,0,...] ) is maybe number of frames.

I'm not good at English, so it could be difficult for you to understand about my write.

ShaoDu commented 5 years ago

Thank you for your reply! I'll have a good look. I'll try again

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"notifications@github.com; 发送时间: 2019年1月29日(星期二) 晚上6:10 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"pytorch-vsumm-reinforce@noreply.github.com; 抄送: "邵都"2577414131@qq.com; "Author"author@noreply.github.com; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15)

No, we should save frames in folder. I told you that read dataset/readme.txt to convert frames from video correctly. Parameter "summary" of frm2video function consists of [0,1,0,1,0,0,0,...]. this length of list( [0,1,0,1,0,0,0,...] ) is maybe number of frames.

I'm not good at English, so it could be difficult for you to understand about my write.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

SinDongHwan commented 5 years ago

if you have dataset video, could you share that? i don't have videos.

ShaoDu commented 5 years ago

All my video data is MP4 video segmented by ffmpeg, but now there is no video data, because the previous deleted data, now there is no operation

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"notifications@github.com; 发送时间: 2019年1月29日(星期二) 晚上6:17 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"pytorch-vsumm-reinforce@noreply.github.com; 抄送: "邵都"2577414131@qq.com; "Author"author@noreply.github.com; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15)

if you have dataset video, could you share that? i don't have videos.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

SinDongHwan commented 5 years ago

okay. i will try to do summary2video, too. if you can convert, please reply. if i will be converted, i reply, too.

SinDongHwan commented 5 years ago

Did you solve the problem? I solved!! I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.

you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to 'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg' ( if extract Summe dataset, there is 'SumMe' directory. )

step 2. convert summe dataset to frames.

i executed 'videos2frames.sh' in 'SumMe/videos' directory. ( you should make each directories about .mp4 as 'SumMe/videos/frames/ For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )

step 3. have to map h5.file key and frame dirctory name

in my case, i made json file. ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }. you can see the information about mapping following code.

import h5py
import sys
h5_file_name = (sys.argv[1])
f = h5py.File(h5_file_name,'r')
for key in f.keys():
    print("{} : {}".format(key, f[key]['video_name'].value))

if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.^^

ShaoDu commented 5 years ago

Thank you very much for your help! Because I am busy with something else now, I will try according to your steps after I finish my work.

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"notifications@github.com; 发送时间: 2019年1月30日(星期三) 上午9:58 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"pytorch-vsumm-reinforce@noreply.github.com; 抄送: "邵都"2577414131@qq.com; "Author"author@noreply.github.com; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15)

Did you solve the problem? I solved!! I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.

you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to 'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg' ( if extract Summe dataset, there is 'SumMe' directory. )

step 2. convert summe dataset to frames.

i executed 'videos2frames.sh' in 'SumMe/videos' directory. ( you should make each directories about .mp4 as 'SumMe/videos/frames/ For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )

step 3. have to map h5.file key and frame dirctory name

in my case, i made json file. ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }. you can see the information about mapping following code. import h5py import sys h5_file_name = (sys.argv[1]) f = h5py.File(h5_file_name,'r') for key in f.keys(): print("{} : {}".format(key, f[key]['video_name'].value))
if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Swati640 commented 5 years ago

Did you solve the problem? I solved!! I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.
* you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to
  'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg'
  ( if extract Summe dataset, there is 'SumMe' directory. )
step 2. convert summe dataset to frames.
* i executed 'videos2frames.sh' in 'SumMe/videos' directory.
  ( you should make each directories about _.mp4 as 'SumMe/videos/frames/_
  For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )
step 3. have to map h5.file key and frame dirctory name
* in my case, i made json file.
  ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }.
  you can see the information about mapping following code.
import h5py
import sys
h5_file_name = (sys.argv[1])
f = h5py.File(h5_file_name,'r')
for key in f.keys():
    print("{} : {}".format(key, f[key]['video_name'].value))
if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.^^

Hello i tried to follow the procedure you told above and made the required changes in shell file as well. But, while running a shell file I am getting "Conversion failed error". This is the error "[image2 @ 0x5654ab990a80] Could not open file : SumMe/videos/frames/Valparaiso_Downhill/%6d.jpg av_interleaved_write_frame(): Input/output error" frame= 1 fps=0.0 q=8.9 Lsize=N/A time=00:00:00.03 bitrate=N/A speed=1.58x
video:79kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Conversion failed!

Please help!! I am stuck with this conversion process.

SinDongHwan commented 5 years ago

@Swati640 Hi, I think can't open file(image). Did you try to edit shell script?

* you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to
  'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg'
  ( if extract Summe dataset, there is 'SumMe' directory. )

If you are okay, i can see your shell on Teamviewer?

Swati640 commented 5 years ago

yes, I edited the shell file completely as you told above, but it is not working.

SinDongHwan commented 5 years ago

@Swati640 hmm... If you are okay, could i see your situation on Teamviewer?

Swati640 commented 5 years ago

Hey.. Thanks for the help. It worked for me. Had to do some changes in the shell file :)

Harryjun commented 5 years ago

Hi, @Swati640 FFMPEG generate frames slowly,Can I take video source as input directly,and generate video summary

Harryjun commented 5 years ago

Hi, @SinDongHwan FFMPEG generate frames slowly,Can I take video source as input directly,and generate video summary

Swati640 commented 5 years ago

you can generate the frames, the way its feasible for you. I used ffmpeg and "https://github.com/SinDongHwan/pytorch-vsumm-reinforce/blob/master/utils/generate_dataset.py" , for generation of frames and feature extraction on my own dataset. I hope i understood your question properly.

SinDongHwan commented 5 years ago

@Harryjun Thank you for kindly answer, instead of me^^

ghagareomkar commented 4 years ago

@SinDongHwan I created a folder named video_frames which contains folder :Air_Force_One,Base_jumping.... and others containing video frames. But still i am getting the same error.Anyone who can help me out. ThankYou

SinDongHwan commented 4 years ago

@ghagareomkar HI, Could you tell me a error message?

ghagareomkar commented 4 years ago

@SinDongHwan Thank You for your reply python summary2video.py -p log/summe-split0/result.h5 -d video_frames/ -i 0 --fps 30 --save-dir log --save-name summary.mp4 this occured： OpenCV: FFMPEG: tag 0x5634504d/'MP4V' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)' OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v' Traceback (most recent call last): File "summary2video.py", line 43, in frm2video(args.frm_dir, summary, vid_writer) File "summary2video.py", line 27, in frm2video frm = cv2.resize(frm, (args.width, args.height)) cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4044: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

This above error i was getting when created a folder named video_frames which contains folder :Air_Force_One,Base_jumping.... and others containing video frames. But when i put just the frames of only one video in video_frames folder .got no error, working properly got summary for the corresponding video .Is my approach correct? ThankYou

SinDongHwan commented 4 years ago

@ghagareomkar please check data of frm before cv2.resize() print(frm)

ghagareomkar commented 4 years ago

@SinDongHwan ThankYou ,will try this

PedroJSousa commented 3 years ago

I went through this whole comment section, and this is super confusing. This is how I managed to do it, I hope it saved some time.

Note: Make sure you have the correct dataset downloaded, because it is not on github, you can download it here Note2: In that dataset the videos2frames.sh file is actually named videos2frames.sh~ You have to change the name of the file by running in the terminal: mv videos2frames.sh~ videos2frames.sh

Step 1: Create a folder called "frames" inside the SumMe/videos folder Step 2:

Install ffmpeg
Edit 2 lines in the videos2frames.sh file: Add a new Line before line 10: mkdir frames/"$name". This will create inside the frames folder a folder with the name of each file
Edit line 11 to this: ffmpeg -i "$f" -f image2 -vf fps=30 frames/"$name"/%6d.jpg. This will generate the frames and save them into correspondent folder. FPS needs to be set to 30 because the results.h5 file is also at 30 fps.

Step3: Run summary2video.py

Check which videos are your test videos. You can see it when you output the results of your training. In my case image below. Note that the videos are numbered. This means e.g. video 25 is the last video in alphabetical order from the dataset: Valparaiso_Downhill
Run the function with the inputs: -p path of the results file ; -d path of where the frames are stored; -i is the index of your training videos matrix. So in my case 0 = video 12, 1 = video 25, 2 = video 4 etc Choose a video from the 5 test videos and generate the highlight. I chose video 25 and ran

summary2video.py -p log/summe-split0/result.h5 -d datasets/videos/frames/Valparaiso_Downhill -i 1 --fps 30 --save-dir log --save-name summary.mp4

sky-16 commented 3 years ago

I went through this whole comment section, and this is super confusing. This is how I managed to do it, I hope it saved some time.

Note: Make sure you have the correct dataset downloaded, because it is not on github, you can download it here Note2: In that dataset the videos2frames.sh file is actually named videos2frames.sh~ You have to change the name of the file by running in the terminal: mv videos2frames.sh~ videos2frames.sh

Step 1: Create a folder called "frames" inside the SumMe/videos folder Step 2:

Install ffmpeg

Edit 2 lines in the videos2frames.sh file: Add a new Line before line 10: mkdir frames/"$name". This will create inside the frames folder a folder with the name of each file

Edit line 11 to this: ffmpeg -i "$f" -f image2 -vf fps=30 frames/"$name"/%6d.jpg. This will generate the frames and save them into correspondent folder. FPS needs to be set to 30 because the results.h5 file is also at 30 fps.

Step3: Run summary2video.py

Check which videos are your test videos. You can see it when you output the results of your training. In my case image below. Note that the videos are numbered. This means e.g. video 25 is the last video in alphabetical order from the dataset: Valparaiso_Downhill

Run the function with the inputs: -p path of the results file ; -d path of where the frames are stored; -i is the index of your training videos matrix. So in my case 0 = video 12, 1 = video 25, 2 = video 4 etc Choose a video from the 5 test videos and generate the highlight. I chose video 25 and ran

summary2video.py -p log/summe-split0/result.h5 -d datasets/videos/frames/Valparaiso_Downhill -i 1 --fps 30 --save-dir log --save-name summary.mp4

I was currently using a dataset of 165 MB mentioned in #40. I was getting error in that dataset during generating summary2video.py and I found out I need to use original Summe dataset of 2.2 GB. Can anyone guide me how to start from beginning. Thanks

SinDongHwan commented 3 years ago

Hi, @sky-16. If you can run teamviewer, send to me email. i will run a code.

wangsen-sen commented 3 years ago

I want to ask, is it necessary to extract all the video frames in the dataset to make a frame dictionary? Or just a few of them? Is it OK to name them in order？

wangsen-sen commented 3 years ago

Did you solve the problem? I solved!! I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.

you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to 'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg' ( if extract Summe dataset, there is 'SumMe' directory. )

step 2. convert summe dataset to frames.

i executed 'videos2frames.sh' in 'SumMe/videos' directory. ( you should make each directories about .mp4 as 'SumMe/videos/frames/ For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )

step 3. have to map h5.file key and frame dirctory name

in my case, i made json file. ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }. you can see the information about mapping following code.
import h5py
import sys
h5_file_name = (sys.argv[1])
f = h5py.File(h5_file_name,'r')
for key in f.keys():
    print("{} : {}".format(key, f[key]['video_name'].value))
if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.^^

excuse me，I want to ask, when I execute 'videos2frames. sh'. 'Processing ./Air_Force_One.webm file... ./Air_Force_One.webm bash: ffmpeg: command not found' What should I do?

SinDongHwan commented 3 years ago

Hi, @wangsen-sen. Install ffmpeg. ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

wangsen-sen commented 3 years ago

Hello，Thank you for your reply. I use the windows system. It shows that the installation has been completed in the CMD. I don't know why there is a problem during the operation.Do you know why？

------------------ 原始邮件 ------------------ 发件人: "KaiyangZhou/pytorch-vsumm-reinforce" @.>; 发送时间: 2021年6月1日(星期二) 中午12:34 @.>; @.**@.>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generate problems (#15)

Hi, @wangsen-sen. Install ffmpeg. ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

wangsen-sen commented 3 years ago

Hi, @wangsen-sen. Install ffmpeg. ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

Thank you very much for your reply. This problem has just been solved. If you have any questions, I'll ask you again.Good luck to you.

wangsen-sen commented 3 years ago

Hi, @wangsen-sen. Install ffmpeg. ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

Hello, excuse me again. I want to ask if the final video summary can be obtained through the file, which frame is included in the original video？

SinDongHwan commented 3 years ago

Hi, @wangsen-sen.

you can check which frames is selected in result.h5 file. selected frames are assigned '1', not selected frames are assigned '0'.

wangsen-sen commented 3 years ago

Hi, @wangsen-sen.

you can check which frames is selected in result.h5 file. selected frames are assigned '1', not selected frames are assigned '0'.

Hello,I processed the result.h5 file and got the final videos tested, but I want to know which frame of the video summary is composed of. Where should I look?

SinDongHwan commented 3 years ago

Hi, @wangsen-sen.

you can check selected frames using following code.

https://github.com/KaiyangZhou/pytorch-vsumm-reinforce/blob/77f7c93249240d6fa297818c1eefd5c2b51b1747/summary2video.py#L41

wangsen-sen commented 3 years ago

Ok，thank you very much.

------------------ 原始邮件 ------------------ 发件人: "KaiyangZhou/pytorch-vsumm-reinforce" @.>; 发送时间: 2021年6月3日(星期四) 上午10:21 @.>; @.**@.>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generate problems (#15)

Hi, @wangsen-sen.

you can check selected frames using following code.

https://github.com/KaiyangZhou/pytorch-vsumm-reinforce/blob/77f7c93249240d6fa297818c1eefd5c2b51b1747/summary2video.py#L41

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

KaiyangZhou / pytorch-vsumm-reinforce

Video can't generate problems #15