Videos at 25fps tend to be choppy by default

[Fritskee]

I'm currently working on a project where I'm utilizing your method. I've observed that the method is configured to operate at 25 frames per second (fps), and I'm trying to understand the rationale behind this choice.

I have a collection of videos in my dataset that are recorded at 30fps, which is a standard frame rate for many recording devices including the smartphone.

However, when I downsample these videos (with ffmpeg) from 30fps to 25fps in order to match the method's operating rate, the resultant videos appear very choppy and lack smoothness. I have tried adding motion blur between the frames, but without much success.

Are there specific reasons why the method is set to work at 25fps instead of the more commonly used 30fps? Would it be possible to modify the method to operate at 30fps without significantly impacting its performance or the results?

Additionally, I would appreciate any suggestions on how to prevent the loss of smoothness when downsampling videos from 30fps to 25fps.

[Inferencer]

I use davinci resolve and set the project and playback rate to 25 fps then when I import my vid I select do not change framerate then I export the results, it's clear to me based on your's and others I have heard from that ffmpeg cannot handle fps change or are using a bad command line. So try with a video editing tool instead.

[Fritskee]

Great advice, I will try a similar approach like yours and see what the result is!

[Fritskee]

@Inferencer @Weizhi-Zhong Updatet on my experiments:

1. I recorded a video with my smartphone at 30fps
2. I used Final Cut Pro to downsample the video to 25fps. When I watch this video it doesn't seem like it's choppy, so this should be a good starting point
3. When I use this video as input, ffmpeg doesn't need to resample, meaning it is 25fps and my export has worked as it should have
4. The resulting output video is still very choppy. I wonder what can be done to try and improve this?

I did also play around with the audio settings. I saw that the method uses 16000Hz for audio, and my original audio sample was recorded at 48000Hz. I have tested the following:

1. Resample my audio sample to 16000Hz and run the method again
2. in audio.py adjusted the settings to 48000Hz. I changed the sampling rate, but also adjusted the window sizes accordingly (n_fft, hop_size, win_size).

Any ideas? I would love to figure this out and hope that we can all learn more from this.

[Fritskee]

@Inferencer @Weizhi-Zhong Updatet on my experiments:

1. I recorded a video with my smartphone at 30fps
2. I used Final Cut Pro to downsample the video to 25fps. When I watch this video it doesn't seem like it's choppy, so this should be a good starting point
3. When I use this video as input, ffmpeg doesn't need to resample, meaning it is 25fps and my export has worked as it should have
4. The resulting output video is still very choppy. I wonder what can be done to try and improve this?

I did also play around with the audio settings. I saw that the method uses 16000Hz for audio, and my original audio sample was recorded at 48000Hz. I have tested the following:

1. Resample my audio sample to 16000Hz and run the method again
2. in audio.py adjusted the settings to 48000Hz. I changed the sampling rate, but also adjusted the window sizes accordingly (n_fft, hop_size, win_size).

Any ideas? I would love to figure this out and hope that we can all learn more from this.

I wanted to add some context: I was initially testing with a clip that took 20 seconds (and the same for my audio recording). Going to a video clip (and audio recording) that only takes 5 seconds seems to improve the issue that I'm facing a lot!

[Inferencer]

Only thing I could suggest is making sure your audio is 25fps too, might help, might not. or swap over to DINet it's a better tool for lip-sync, providing your video has studio lighting & is facing the camera I have 4gb GPU and can do 1920x1080 vids with it https://streamable.com/s6wgqr

[Fritskee]

Only thing I could suggest is making sure your audio is 25fps too, might help, might not. or swap over to DINet it's a better tool for lip-sync, providing your video has studio lighting & is facing the camera I have 4gb GPU and can do 1920x1080 vids with it https://streamable.com/s6wgqr

Oh wow, that's a very nice result with the video you show me. I will try this one out as well! Thanks a lot @Inferencer!

edit: Also: @Inferencer I looked at your starred repos, seems like we have a lot of interests in common. Let me know if you would like to connect.

[Inferencer]

Only thing I could suggest is making sure your audio is 25fps too, might help, might not. or swap over to DINet it's a better tool for lip-sync, providing your video has studio lighting & is facing the camera I have 4gb GPU and can do 1920x1080 vids with it https://streamable.com/s6wgqr

Oh wow, that's a very nice result with the video you show me. I will try this one out as well! Thanks a lot @Inferencer!

edit: Also: @Inferencer I looked at your starred repos, seems like we have a lot of interests in common. Let me know if you would like to connect.

sure thing, always good to share tips etc I'm on Discord @inferencer (same name as github)

[Fritskee]

Only thing I could suggest is making sure your audio is 25fps too, might help, might not. or swap over to DINet it's a better tool for lip-sync, providing your video has studio lighting & is facing the camera I have 4gb GPU and can do 1920x1080 vids with it https://streamable.com/s6wgqr

Oh wow, that's a very nice result with the video you show me. I will try this one out as well! Thanks a lot @Inferencer! edit: Also: @Inferencer I looked at your starred repos, seems like we have a lot of interests in common. Let me know if you would like to connect.

sure thing, always good to share tips etc I'm on Discord @Inferencer (same name as github)

Cool, I sent you a request!