search

continue-revolution / sd-webui-animatediff

AnimateDiff for AUTOMATIC1111 Stable Diffusion WebUI
Other
3.1k stars 258 forks source link

[Bug]: Vid2Vid issues #465

Closed DavideAlidosi closed 7 months ago

DavideAlidosi commented 8 months ago

Is there an existing issue for this?

Have you read FAQ on README?

What happened?

From the tests I have conducted, it appears that Vid2Vid's system continues to be plagued by malfunctions. Currently, proper conversion cannot be achieved, for example, with Line Art and other controls. Currently this is my configuration: AnimateDiff 2.0.0 ControlNet 1.441 WebUI 1.8.0 Everything has been updated as of today.

Steps to reproduce the problem

  1. Enable AnimateDiff
  2. Model: mm_sd15_v3.safetensors
  3. Load up a video .mp4
  4. Enable ControlNet unit 0
  5. Select Lineart
  6. Click on generate

What should have happened?

Normally it would start converting the video, but instead only the first 3 frames are loaded and the result unserviceable.

Commit where the problem happens

webui: 1.8.0 extension: 2.0.0

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--xformers

Console logs

2024-03-14 15:39:55,091 - AnimateDiff - INFO - AnimateDiff process start.
2024-03-14 15:39:55,091 - AnimateDiff - INFO - Loading motion module mm_sd15_v3.safetensors from L:\WebUI\webui\extensions\sd-webui-animatediff\model\mm_sd15_v3.safetensors
2024-03-14 15:39:55,130 - AnimateDiff - INFO - Guessed mm_sd15_v3.safetensors architecture: MotionModuleType.AnimateDiffV3
2024-03-14 15:39:58,659 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet input blocks.
2024-03-14 15:39:58,659 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet output blocks.
2024-03-14 15:39:58,659 - AnimateDiff - INFO - Setting DDIM alpha.
2024-03-14 15:39:58,666 - AnimateDiff - INFO - Injection finished.
2024-03-14 15:39:58,666 - AnimateDiff - INFO - Attempting to extract frames via ffmpeg from C:\Users\indrema\AppData\Local\Temp\gradio\4d7028214e22e57e767c24b1b4fd4b7e3817cbe1\video_demo_16frame.mp4 to L:\WebUI\webui/tmp/animatediff-frames\video_demo_16frame-ab70ed73
ffmpeg version 2023-11-22-git-0008e1c5d5-full_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --pkg-config=pkgconf --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-dxva2 --enable-d3d11va --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      58. 32.100 / 58. 32.100
  libavcodec     60. 34.100 / 60. 34.100
  libavformat    60. 17.100 / 60. 17.100
  libavdevice    60.  4.100 / 60.  4.100
  libavfilter     9. 13.100 /  9. 13.100
  libswscale      7.  6.100 /  7.  6.100
  libswresample   4. 13.100 /  4. 13.100
  libpostproc    57.  4.100 / 57.  4.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\Users\indrema\AppData\Local\Temp\gradio\4d7028214e22e57e767c24b1b4fd4b7e3817cbe1\video_demo_16frame.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41
    creation_time   : 2024-02-12T10:19:39.000000Z
  Duration: 00:00:00.53, start: 0.000000, bitrate: 14484 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 576x1024, 13303 kb/s, 30 fps, 30 tbr, 30k tbn (default)
    Metadata:
      creation_time   : 2024-02-12T10:19:40.000000Z
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : AVC Coding
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 248 kb/s (default)
    Metadata:
      creation_time   : 2024-02-12T10:19:40.000000Z
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[swscaler @ 000001edac1e6480] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 000001edd013ff80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 000001edac1e6480] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 1 times
Output #0, image2, to 'L:\WebUI\webui\tmp\animatediff-frames\video_demo_16frame-ab70ed73\%09d.jpg':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41
    encoder         : Lavf60.17.100
  Stream #0:0(eng): Video: mjpeg, yuvj420p(pc, bt709, progressive), 576x1024, q=2-31, 200 kb/s, 30 fps, 30 tbn (default)    Metadata:
      creation_time   : 2024-02-12T10:19:40.000000Z
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.34.100 mjpeg
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
[out#0/image2 @ 000001edabaf48c0] video:215kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=    3 fps=0.0 q=1.0 Lsize=N/A time=00:00:00.06 bitrate=N/A speed=0.781x
2024-03-14 15:39:59,121 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 3 frames.
2024-03-14 15:39:59,160 - ControlNet - INFO - unit_separate = False, style_align = False
2024-03-14 15:39:59,358 - ControlNet - INFO - Loading model: control_v11p_sd15_lineart [43d4be0d]
2024-03-14 15:40:00,010 - ControlNet - INFO - Loaded state_dict from [L:\WebUI\webui\models\ControlNet\control_v11p_sd15_lineart.pth]
2024-03-14 15:40:00,011 - ControlNet - INFO - controlnet_default_config
2024-03-14 15:40:02,670 - ControlNet - INFO - ControlNet model control_v11p_sd15_lineart [43d4be0d](ControlModelType.ControlNet) loaded.
2024-03-14 15:40:02,671 - ControlNet - INFO - AnimateDiff + ControlNet lineart receive the following parameters:
2024-03-14 15:40:02,671 - ControlNet - INFO -   batch control images: L:\WebUI\webui/tmp/animatediff-frames\video_demo_16frame-ab70ed73
2024-03-14 15:40:02,693 - ControlNet - INFO - Using preprocessor: lineart
2024-03-14 15:40:02,693 - ControlNet - INFO - preprocessor resolution = 512
100%|████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 10.95it/s]
2024-03-14 15:40:03,026 - ControlNet - INFO - ControlNet Hooked - Time = 3.87398624420166
  0%|                                                                                           | 0/20 [00:00<?, ?it/s]2024-03-14 15:40:03,453 - AnimateDiff - INFO - inner model forward hooked
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:07<00:00,  2.63it/s]
2024-03-14 15:40:11,541 - AnimateDiff - INFO - Restoring DDIM alpha.███████████████████| 20/20 [00:05<00:00,  3.51it/s]
2024-03-14 15:40:11,541 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet input blocks.
2024-03-14 15:40:11,542 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet output blocks.
2024-03-14 15:40:11,543 - AnimateDiff - INFO - Removal finished.
2024-03-14 15:40:11,544 - AnimateDiff - INFO - Saving output formats: PNG, WEBP
2024-03-14 15:40:11,761 - AnimateDiff - INFO - Saving output\txt2img-images\AnimateDiff\2024-03-14\00017-2059008824.webp with lossless=True and quality=80
2024-03-14 15:40:11,832 - AnimateDiff - INFO - AnimateDiff process end.
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00,  3.15it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00,  3.51it/s]

Additional information

No response

continue-revolution commented 8 months ago

I don't know why but ffmpeg only extracts the first 3 frames. Please take a look at L:\WebUI\webui/tmp/animatediff-frames\video_demo_16frame-ab70ed73 in your filesystem and see if there is really only 3 frames extracted.

I will add an option to disable ffmpeg later today and let you use cv2 by default.

continue-revolution commented 7 months ago

468 I've added an option to let you choosing opencv as your frame extraction method. please switch to opencv in Settings/AnimateDiff and retry. For now I will close this issue, but if it does not work, please feel free to re-open it and let me know.

DavideAlidosi commented 7 months ago

Thank you very much, now everything is finally working perfectly. I can't thank you enough. Bests.