search

avinashpaliwal / Super-SloMo

PyTorch implementation of Super SloMo by Jiang et al.
MIT License
2.99k stars 476 forks source link

Issue with the output when evaluating with the pretrained model #7

Closed ybabs closed 4 years ago

ybabs commented 5 years ago

Hi, forgive me if this sounds a bit dense, I don't really know anything about machine learning but I have a passing interest in making slomo videos. I'm currently evaluating on a CPU (AMD Threradripper 1920x) as I don't have a CUDA device but I'm having trouble with the output from the video_to_slomo.py script. As a reference, I used ffmpeg to convert your original gif to mp4 and tried to create a video but the output looks a bit off. This is what the converted video looks like: Link here. Another video I tried also looks like this. Any ideas why this is?

Thanks

avinashpaliwal commented 5 years ago

That is weird. The channel wise mean is subtracted from the frames before passing them as input and added back to the output. Maybe something went wrong with that. Can you post your output after running the script. Also mention the pytorch version you have installed.

ybabs commented 5 years ago

pytorch version is 1.0.0

This is my output after running the script:

ffmpeg version 4.0.3-1~18.04.york0 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version='1~18.04.york0' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version='1~18.04.york0' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/homepc/Super-SloMo/input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.12.100
  Duration: 00:00:01.24, start: 0.000000, bitrate: 948 kb/s
    Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 480x270 [SAR 1:1 DAR 16:9], 940 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[swscaler @ 0x55f8c904c7c0] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to 'tmpSuperSloMo/input/%06d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.12.100
    Stream #0:0(und): Video: mjpeg, yuvj444p(pc), 480x270 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.18.100 mjpeg
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
frame=   31 fps=0.0 q=2.0 Lsize=N/A time=00:00:01.24 bitrate=N/A speed=25.7x    
video:1452kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
/home/homepc/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode))
/home/homepc/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:1332: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
  warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
/usr/bin/ffmpeg -r 240.0 -i tmpSuperSloMo/output/%d.jpg -qscale:v 2 output.mp4
ffmpeg version 4.0.3-1~18.04.york0 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version='1~18.04.york0' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version='1~18.04.york0' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
Input #0, image2, from 'tmpSuperSloMo/output/%d.jpg':
  Duration: 00:00:04.80, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 480x270 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x556ca73d8000] -qscale is ignored, -crf is recommended.
[libx264 @ 0x556ca73d8000] using SAR=1/1
[libx264 @ 0x556ca73d8000] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x556ca73d8000] profile High, level 3.2
[libx264 @ 0x556ca73d8000] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=8 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf58.12.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuvj420p(pc), 480x270 [SAR 1:1 DAR 16:9], q=-1--1, 240 fps, 15360 tbn, 240 tbc
    Metadata:
      encoder         : Lavc58.18.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 3 times
[swscaler @ 0x556ca7936280] deprecated pixel format used, make sure you did set range correctly
Past duration 8.999992 too large
    Last message repeated 2 times
frame=  129 fps=0.0 q=-1.0 Lsize=     408kB time=00:00:00.52 bitrate=6368.6kbits/s dup=9 drop=0 speed=0.931x    
video:406kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.577492%
[libx264 @ 0x556ca73d8000] frame I:1     Avg QP:27.57  size: 19262
[libx264 @ 0x556ca73d8000] frame P:38    Avg QP:31.32  size:  3687
[libx264 @ 0x556ca73d8000] frame B:90    Avg QP:34.64  size:  2839
[libx264 @ 0x556ca73d8000] consecutive B-frames:  0.8% 18.6%  0.0% 80.6%
[libx264 @ 0x556ca73d8000] mb I  I16..4:  0.4% 94.3%  5.3%
[libx264 @ 0x556ca73d8000] mb P  I16..4:  0.4%  1.9%  1.4%  P16..4: 49.2% 20.6% 11.3%  0.0%  0.0%    skip:15.2%
[libx264 @ 0x556ca73d8000] mb B  I16..4:  0.8%  3.3%  2.3%  B16..8: 18.8%  5.5%  2.1%  direct:34.0%  skip:33.3%  L0:80.0% L1:12.5% BI: 7.5%
[libx264 @ 0x556ca73d8000] 8x8 transform intra:56.9% inter:69.6%
[libx264 @ 0x556ca73d8000] coded y,uvDC,uvAC intra: 80.4% 49.8% 21.2% inter: 46.6% 11.3% 3.2%
[libx264 @ 0x556ca73d8000] i16 v,h,dc,p: 29% 45%  6% 21%
[libx264 @ 0x556ca73d8000] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 19% 20%  6%  5%  7%  6%  7%  9%
[libx264 @ 0x556ca73d8000] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 35% 14%  6%  5%  6%  6%  5%  7%
[libx264 @ 0x556ca73d8000] i8c dc,h,v,p: 75% 15%  9%  2%
[libx264 @ 0x556ca73d8000] Weighted P-Frames: Y:55.3% UV:34.2%
[libx264 @ 0x556ca73d8000] ref P L0: 53.4% 16.6%  3.5% 24.8%  1.6%
[libx264 @ 0x556ca73d8000] ref B L0: 40.2% 59.7%  0.0%
[libx264 @ 0x556ca73d8000] ref B L1: 55.3% 44.7%
[libx264 @ 0x556ca73d8000] kb/s:6175.27
avinashpaliwal commented 5 years ago

I am getting the same issue when running on CPU. I am working on it.

avinashpaliwal commented 5 years ago

I have added a temporary fix to the script. You can try running it now.

ybabs commented 5 years ago

That fixed it thanks.

YaoooLiang commented 5 years ago

Hi, @avinashpaliwal @InterestingWalrus When I run the command below, the interpolation results are all copies of the previous frame.This is also found in other tests of video. Any ideas why this is?

python video_to_slomo.py --video misc/original.gif --sf 2 --checkpoint ./SuperSloMo.ckpt --fps 30 --output videos/out_gif.mp4

avinashpaliwal commented 5 years ago

Can you post more details. Like cmd/terminal output of script. Post some example input and output video as well. Just to clarify, you are getting output frames where every interpolated frame is the copy of the original frame before it?

YaoooLiang commented 5 years ago

This is my output after running the script:

[root@7a382a96-08ca-454d-b42c-d1a554518421 Super-SloMo]# python video_to_slomo.py --video misc/original.gif --sf 2 --checkpoint ./SuperSloMo.ckpt --fps 30 --output videos/out_gif.mp4 ffmpeg -i misc/original.gif -vsync 0 -qscale:v 2 tmpSuperSloMo/input/%06d.jpg ffmpeg version 2.6.8 Copyright (c) 2000-2016 the FFmpeg developers built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-4) configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-gnutls --enable-ladspa --enable-libass --enable-libcdio --enable-libdc1394 --enable-libfaac --enable-nonfree --enable-libfdk-aac --enable-nonfree --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libx264 --enable-libx265 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect libavutil 54. 20.100 / 54. 20.100 libavcodec 56. 26.100 / 56. 26.100 libavformat 56. 25.101 / 56. 25.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 11.102 / 5. 11.102 libavresample 2. 1. 0 / 2. 1. 0 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 1.100 / 1. 1.100 libpostproc 53. 3.100 / 53. 3.100 Input #0, gif, from 'misc/original.gif': Duration: N/A, bitrate: N/A Stream #0:0: Video: gif, bgra, 480x270, 25 fps, 25 tbr, 100 tbn, 100 tbc [swscaler @ 0x13b97e0] deprecated pixel format used, make sure you did set range correctly Output #0, image2, to 'tmpSuperSloMo/input/%06d.jpg': Metadata: encoder : Lavf56.25.101 Stream #0:0: Video: mjpeg, yuvj444p(pc), 480x270, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc Metadata: encoder : Lavc56.26.100 mjpeg Stream mapping: Stream #0:0 -> #0:0 (gif (native) -> mjpeg (native)) Press [q] to stop, [?] for help frame= 31 fps=0.0 q=0.0 Lsize=N/A time=00:00:01.24 bitrate=N/A
video:1906kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown /usr/local/lib/python2.7/site-packages/torch/nn/functional.py:1961: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)) /usr/local/lib/python2.7/site-packages/torch/nn/functional.py:1006: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead. warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.") ffmpeg -r 30.0 -i tmpSuperSloMo/output/%d.jpg -qscale:v 2 videos/out_gif.mp4 ffmpeg version 2.6.8 Copyright (c) 2000-2016 the FFmpeg developers built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-4) configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-gnutls --enable-ladspa --enable-libass --enable-libcdio --enable-libdc1394 --enable-libfaac --enable-nonfree --enable-libfdk-aac --enable-nonfree --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libx264 --enable-libx265 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect libavutil 54. 20.100 / 54. 20.100 libavcodec 56. 26.100 / 56. 26.100 libavformat 56. 25.101 / 56. 25.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 11.102 / 5. 11.102 libavresample 2. 1. 0 / 2. 1. 0 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 1.100 / 1. 1.100 libpostproc 53. 3.100 / 53. 3.100 [mjpeg @ 0x1fe2e20] Changeing bps to 8 Input #0, image2, from 'tmpSuperSloMo/output/%d.jpg': Duration: 00:00:02.40, start: 0.000000, bitrate: N/A Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 480x270 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc File 'videos/out_gif.mp4' already exists. Overwrite ? [y/N] y No pixel format specified, yuvj420p for H.264 encoding chosen. Use -pix_fmt yuv420p for compatibility with outdated media players. [libx264 @ 0x2011300] -qscale is ignored, -crf is recommended. [libx264 @ 0x2011300] using SAR=1/1 [libx264 @ 0x2011300] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2 [libx264 @ 0x2011300] profile High, level 2.1 [libx264 @ 0x2011300] 264 - core 142 r2495 6a301b6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=72 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 Output #0, mp4, to 'videos/out_gif.mp4': Metadata: encoder : Lavf56.25.101 Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuvj420p(pc), 480x270 [SAR 1:1 DAR 16:9], q=-1--1, 30 fps, 15360 tbn, 30 tbc Metadata: encoder : Lavc56.26.100 libx264 Stream mapping: Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264)) Press [q] to stop, [?] for help frame= 60 fps=0.0 q=-1.0 Lsize= 322kB time=00:00:01.93 bitrate=1362.6kbits/s
video:320kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.474768% [libx264 @ 0x2011300] frame I:1 Avg QP:26.12 size: 21812 [libx264 @ 0x2011300] frame P:30 Avg QP:26.46 size: 9479 [libx264 @ 0x2011300] frame B:29 Avg QP:30.68 size: 719 [libx264 @ 0x2011300] consecutive B-frames: 3.3% 96.7% 0.0% 0.0% [libx264 @ 0x2011300] mb I I16..4: 0.0% 98.0% 2.0% [libx264 @ 0x2011300] mb P I16..4: 0.0% 3.2% 0.3% P16..4: 31.3% 35.2% 26.2% 0.0% 0.0% skip: 3.7% [libx264 @ 0x2011300] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 45.5% 5.4% 1.3% direct: 1.3% skip:46.4% L0:59.2% L1: 1.5% BI:39.3% [libx264 @ 0x2011300] 8x8 transform intra:94.5% inter:78.5% [libx264 @ 0x2011300] coded y,uvDC,uvAC intra: 97.4% 80.9% 24.7% inter: 38.8% 22.3% 2.5% [libx264 @ 0x2011300] i16 v,h,dc,p: 0% 100% 0% 0% [libx264 @ 0x2011300] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 20% 25% 6% 5% 6% 5% 8% 11% [libx264 @ 0x2011300] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 39% 13% 7% 6% 5% 4% 7% 6% [libx264 @ 0x2011300] i8c dc,h,v,p: 61% 20% 16% 3% [libx264 @ 0x2011300] Weighted P-Frames: Y:3.3% UV:0.0% [libx264 @ 0x2011300] ref P L0: 68.7% 16.1% 12.0% 3.0% 0.1% [libx264 @ 0x2011300] ref B L0: 99.8% 0.2% [libx264 @ 0x2011300] kb/s:1308.20

YaoooLiang commented 5 years ago

Input video is your original gif ; this is my output of video :out_gif.zip

avinashpaliwal commented 5 years ago

I am getting correct results on my end. The video you have posted has no interpolated frames, they look like exact copies of previous frames. I haven't tested the whole thing on linux (I am using windows), so I will try that. In the mean time try running it again with configuration pytorch 0.4.1, python 3.6. Comment out this line in the script and check the hidden folder tmpSuperSloMo for the generated output frames. There will be a resolution mismatch which I am fixing, so ignore that.

ybabs commented 5 years ago

@YaoooLiang @avinashpaliwal, if I don't specify the ffmpeg directory, the script doesn't run. Not sure why it's running for you. Also you seem to be using an old version of ffmpeg. Try updating ffmpeg maybe?

YaoooLiang commented 5 years ago

@InterestingWalrus @avinashpaliwal the hidden folder tmpSuperSloMo was existed, so ffmpeg worked. I had checked tmpSuperSloMo, the interpolated frames in output had a very little change from previous frame. the content in tmpSuperSloMo is : input.zip output.zip

YaoooLiang commented 5 years ago

@YaoooLiang @avinashpaliwal, if I don't specify the ffmpeg directory, the script doesn't run. Not sure why it's running for you. Also you seem to be using an old version of ffmpeg. Try updating ffmpeg maybe?

Because I had changed extract_frames and create_video code , so it was worked for me .

avinashpaliwal commented 5 years ago

@YaoooLiang @avinashpaliwal, if I don't specify the ffmpeg directory, the script doesn't run. Not sure why it's running for you. Also you seem to be using an old version of ffmpeg. Try updating ffmpeg maybe?

Like I said, I am using windows. The ffmpeg comes in a zip which you have to extract. That's why you have to provide the path for ffmpeg.exe if it's not in your project folder. I have not tested the project on linux, so I don't know the problems yet. I will work on that, but it will take some time.

avinashpaliwal commented 5 years ago

@YaoooLiang @avinashpaliwal, if I don't specify the ffmpeg directory, the script doesn't run. Not sure why it's running for you. Also you seem to be using an old version of ffmpeg. Try updating ffmpeg maybe?

Because I had changed extract_frames and create_video code , so it was worked for me .

I don't know why you are getting such results. I will have to test the project on linux. Also, you said you modified the code, so did you also modify the code in section where frame is interpolated. Post your modified script so I can run it on my end.

YaoooLiang commented 5 years ago

@avinashpaliwal My test results on Windows are exactly the same as yours, but I don't know why the results on Linux are wrong

Godatplay commented 5 years ago

I was able to get this working just fine on Linux myself. I'm on Peppermint (Ubuntu-based), using PyTorch 0.4.1 with this conda build: py37_cuda9.2.148_cudnn7.1.4_1

I had to uninstall ffmpeg from conda because the included package doesn't include libx264, so I used apt to install it plus the libavcodec-extra metapackage.

My test video is here and the results can be found here. They seem to line up with the poor results examples, but you can see it working well on the mans' pants, background environment, and smaller figures walking in the background. My assumption is that the model needs more data from overlapping biped movement and limb rotation perpendicular to the camera (potentially more non-Caucasian data as well?), but is otherwise working as expected.

deng-wei commented 5 years ago

Hi, @avinashpaliwal I have got the same results as @YaoooLiang in Ubuntu16, the interpolation results are copies of the previous frame. Unfortunately, I don't have a Windows machine. Are there any updates about this? Thank you very much.

avinashpaliwal commented 5 years ago

Hi, @avinashpaliwal I have got the same results as @YaoooLiang in Ubuntu16, the interpolation results are copies of the previous frame. Unfortunately, I don't have a Windows machine. Are there any updates about this? Thank you very much.

I am currently busy with other tasks and am unable to take out time to test the script on Linux. Try following Godatplay's approach and see if it works. I will post an update about Linux after sometime.

Godatplay commented 5 years ago

Specifically I'm on Peppermint 9, which is based on Ubuntu 18.04 LTS, but my guess is that this is an ffmpeg issue. I wanted to maintain the same 25 FPS as the input video, but 4x longer, so I used --sf 4 and --fps 25

jwrona commented 5 years ago

Specifically I'm on Peppermint 9, which is based on Ubuntu 18.04 LTS, but my guess is that this is an ffmpeg issue. I wanted to maintain the same 25 FPS as the input video, but 4x longer, so I used --sf 4 and --fps 25

I don't think this is an ffmpeg issue. The input frames in .tmpSuperSlomo/input are fine, but the interpolated output frames in .tmpSuperSlomo/output are already defective.

avinashpaliwal commented 5 years ago

Specifically I'm on Peppermint 9, which is based on Ubuntu 18.04 LTS, but my guess is that this is an ffmpeg issue. I wanted to maintain the same 25 FPS as the input video, but 4x longer, so I used --sf 4 and --fps 25

I don't think this is an ffmpeg issue. The input frames in .tmpSuperSlomo/input are fine, but the interpolated output frames in .tmpSuperSlomo/output are already defective.

Right. It is mostly likely something to do with the pytorch library on linux. I also used conda, so maybe installing pytorch from some other source is the issue.

jwrona commented 5 years ago

I used conda too. I've tried pytorch 0.4.1 and the current version, both CPU, but the results were the same.

Godatplay commented 5 years ago

Ah ok. I guess I was using GPU on Linux, so likely a pytorch CPU-specific issue.

jwrona commented 5 years ago

Can confirm that this is a pytorch CPU-specific issue. Works fine using current pytorch and CUDA 9.0. CPU: cpu CUDA: cuda

avinashpaliwal commented 5 years ago

Can confirm that this is a pytorch CPU-specific issue. Works fine using current pytorch and CUDA 9.0. CPU: cpu CUDA: cuda

That CPU result sholdn't be like this after the temporary fix. Are you using the latest commit?

jwrona commented 5 years ago

That CPU result sholdn't be like this after the temporary fix. Are you using the latest commit?

Yes. I was using commit e8508bd949e51985ba73b99d0063090cec7e3679

lxlee commented 5 years ago

Hi, @avinashpaliwal I have got the same results as @YaoooLiang in Ubuntu16, the interpolation results are copies of the previous frame. Unfortunately, I don't have a Windows machine. Are there any updates about this? Thank you very much.

Were you running with python2 in linux?

deng-wei commented 5 years ago

Hi, @avinashpaliwal I have got the same results as @YaoooLiang in Ubuntu16, the interpolation results are copies of the previous frame. Unfortunately, I don't have a Windows machine. Are there any updates about this? Thank you very much.

Were you running with python2 in linux?

Yes

lxlee commented 5 years ago

Hi, @avinashpaliwal I have got the same results as @YaoooLiang in Ubuntu16, the interpolation results are copies of the previous frame. Unfortunately, I don't have a Windows machine. Are there any updates about this? Thank you very much.

Were you running with python2 in linux?

Yes

I had met the same problem. The script video_to_slomo.py is not compatibility with python2. It seems that Using python3 instead as suggested or modifying the script both make sense.

arskama commented 5 years ago

Hi, @avinashpaliwal

The fix for cpu color issues is the following:

diff --git a/video_to_slomo.py b/video_to_slomo.py index ae021db..5586a12 100755 --- a/video_to_slomo.py +++ b/video_to_slomo.py @@ -125,7 +125,7 @@ def main():

 # Temporary fix for issue #7 https://github.com/avinashpaliwal/Super-SloMo/issues/7 -
 # - Removed per channel mean subtraction for CPU.

If you debug, you ll see that it falls always in the gpu path, because the if statement is not correct. Sorry for not making a push request on that one.

e-sha commented 5 years ago

Hi, @avinashpaliwal I have got the same results as @YaoooLiang in Ubuntu16, the interpolation results are copies of the previous frame. Unfortunately, I don't have a Windows machine. Are there any updates about this? Thank you very much.

Were you running with python2 in linux?

Yes

I had met the same problem. The script video_to_slomo.py is not compatibility with python2. It seems that Using python3 instead as suggested or modifying the script both make sense.

The problem with python2 was in integer division. It was fixed in one of the pull requests.

rsjjdesj commented 3 years ago

Hi, @avinashpaliwal

The fix for cpu color issues is the following:

diff --git a/video_to_slomo.py b/video_to_slomo.py index ae021db..5586a12 100755 --- a/video_to_slomo.py +++ b/video_to_slomo.py @@ -125,7 +125,7 @@ def main():

 # Temporary fix for issue #7 https://github.com/avinashpaliwal/Super-SloMo/issues/7 -
 # - Removed per channel mean subtraction for CPU.

  • if (device == "cpu"):

  • if (device.type == "cpu"): transform = transforms.Compose([transforms.ToTensor()]) TP = transforms.Compose([transforms.ToPILImage()]) else:

If you debug, you ll see that it falls always in the gpu path, because the if statement is not correct. Sorry for not making a push request on that one.

Can this be pushed in the repository. This should fix the bug.