search

light-and-ray / sd-webui-replacer

A tab for sd-webui for replacing objects in pictures or videos using detection prompt
191 stars 11 forks source link

Memory and FFMPEG #74

Closed niotsujj closed 3 months ago

niotsujj commented 3 months ago 
*** CUDA out of memory. Tried to allocate 6.75 GiB. GPU 0 has a total capacty of 11.99 GiB of which 0 bytes is free. Of the allocated memory 18.76 GiB is allocated by PyTorch, and 6.79 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF ***
    Traceback (most recent call last):
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\video_animatediff.py", line 132, in animatediffGenerate
        processed = processFragment(fragmentPath, initImage, gArgs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\video_animatediff.py", line 25, in processFragment
        processed, _ = inpaint(initImage, gArgs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\inpaint.py", line 95, in inpaint
        processed = process_images(p)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\processing.py", line 845, in process_images
        res = process_images_inner(p)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 48, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\processing.py", line 915, in process_images_inner
        p.init(p.all_prompts, p.all_seeds, p.all_subseeds)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\processing.py", line 1732, in init
        self.image_conditioning = self.img2img_image_conditioning(image * 2 - 1, self.init_latent, image_mask, self.mask_round)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\processing.py", line 388, in img2img_image_conditioning
        return self.inpainting_image_conditioning(source_image, latent_image, image_mask=image_mask, round_image_mask=round_image_mask)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\processing.py", line 366, in inpainting_image_conditioning
        conditioning_image = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(conditioning_image))
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\sd_hijack_utils.py", line 18, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\sd_hijack_utils.py", line 32, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 830, in encode_first_stage
        return self.first_stage_model.encode(x)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\lowvram.py", line 67, in first_stage_model_encode_wrap
        return first_stage_model_encode(x)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 83, in encode
        h = self.encoder(x)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 526, in forward
        h = self.down[i_level].block[i_block](hs[-1], temb)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 138, in forward
        h = self.norm2(h)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions-builtin\Lora\networks.py", line 533, in network_GroupNorm_forward
        return originals.GroupNorm_forward(self, input)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\modules\normalization.py", line 279, in forward
        return F.group_norm(
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\torch\nn\functional.py", line 2558, in group_norm
        return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled)
    torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 6.75 GiB. GPU 0 has a total capacty of 11.99 GiB of which 0 bytes is free. Of the allocated memory 18.76 GiB is allocated by PyTorch, and 6.79 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

---
merging fragments
0it [00:00, ?it/s]
generate done, generating video
C:\Users\justi\Desktop\subt\stable\sd.webui\system\python\lib\site-packages\imageio_ffmpeg\binaries\ffmpeg-win64-v4.2.2.exe -framerate 4 -i C:\Users\justi\Desktop\subt\stable\sd.webui\webui\out_3665889838_1715926658\resultFrames\%5d-3665889838.png -r 4 -i C:\Users\justi\Desktop\subt\stable\sd.webui\webui\7047810717822635266.mp4 -map 0:v:0 -map 1:a:0? -c:v libx264 -c:a aac -vf fps=4 -profile:v main -pix_fmt yuv420p -shortest -y C:\Users\justi\Desktop\subt\stable\sd.webui\webui\out_3665889838_1715926658\output_7047810717822635266.mp4_3665889838_1715926658.mp4
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.1 (GCC) 20200122
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[image2 @ 000001d1c5babd40] Could find no file with path 'C:\Users\justi\Desktop\subt\stable\sd.webui\webui\out_3665889838_1715926658\resultFrames\%5d-3665889838.png' and index in the range 0-4
C:\Users\justi\Desktop\subt\stable\sd.webui\webui\out_3665889838_1715926658\resultFrames\%5d-3665889838.png: No such file or directory
*** Error completing request
*** Arguments: ('task(e83z6r8co54ej7v)', 'tab_batch_video', 't-shirt', '', 'RAW image of a nude woman, naked', '((clothing), (monochrome:1.3), (deformed, distorted, disfigured:1.3), (hair), tattoo, wet, water, clothing, shadow, 3d render, cartoon, ((blurry)), duplicate, ((duplicate body parts)), (disfigured), (poorly drawn), ((missing limbs)), logo, signature, text, words, low res, boring, artifacts, bad art, gross, ugly, poor quality, low quality, poorly drawn, bad anatomy, wrong anatomy', None, None, True, '', '', True, False, 'C:\\Users\\justi\\Desktop\\subt\\stable\\sd.webui\\webui\\7047810717822635266.mp4', '', 4, 'None', -1, 'DPM++ 2M SDE', 'Automatic', 20, 0.3, 35, 4, 1280, 'mobile_sam.pt', 'GroundingDINO_SwinT_OGC (694MB)', 5.5, 1, 40, 0, 512, 512, 1, 1, 0, ['script'], False, False, 'uberRealisticPornMerge_urpmv13Inpainting.safetensors [fe06753eee]', 'Random', ['Draw mask'], None, True, ['Draw mask'], None, True, None, '', 1, False, False, [], False, 'video_mode_animatediff', '-', 0, 0, 4, 1, -1, 1, 16, False, 'control_v11p_sd15_inpaint_fp16 [be8bc0ed]', 1, True, 'uberRealisticPornMerge_urpmv13Inpainting.safetensors [fe06753eee]', 'mm_sd15_v3.safetensors', 'ESRGAN_4x', 4, 'Use same sampler', 'Use same scheduler', 0.35, 1, '', 1800, 'Lanczos', True, True, 5, '', '', 'Use same checkpoint', 250, 2, True, 'Same', ControlNetUnit(is_ui=True, input_mode=<InputMode.SIMPLE: 'simple'>, batch_images='', output_dir='', loopback=False, enabled=False, module='none', model='mm_sd15_v3_sparsectrl_scribble [60d136f2]', weight=1.0, image=None, resize_mode=<ResizeMode.INNER_FIT: 'Crop and Resize'>, low_vram=False, processor_res=-1, threshold_a=-1.0, threshold_b=-1.0, guidance_start=0.0, guidance_end=1.0, pixel_perfect=False, control_mode=<ControlMode.BALANCED: 'Balanced'>, inpaint_crop_input_image=True, hr_option=<HiResFixOption.BOTH: 'Both'>, save_detected_map=True, advanced_weighting=None, effective_region_mask=None, pulid_mode=<PuLIDMode.FIDELITY: 'Fidelity'>, ipadapter_input=None, mask=None, batch_mask_dir=None, animatediff_batch=False, batch_modifiers=[], batch_image_files=[]), ControlNetUnit(is_ui=True, input_mode=<InputMode.SIMPLE: 'simple'>, batch_images='', output_dir='', loopback=False, enabled=False, module='none', model='mm_sd15_v3_sparsectrl_rgb [9f17d62f]', weight=1.0, image=None, resize_mode=<ResizeMode.INNER_FIT: 'Crop and Resize'>, low_vram=False, processor_res=-1, threshold_a=-1.0, threshold_b=-1.0, guidance_start=0.0, guidance_end=1.0, pixel_perfect=False, control_mode=<ControlMode.BALANCED: 'Balanced'>, inpaint_crop_input_image=True, hr_option=<HiResFixOption.BOTH: 'Both'>, save_detected_map=True, advanced_weighting=None, effective_region_mask=None, pulid_mode=<PuLIDMode.FIDELITY: 'Fidelity'>, ipadapter_input=None, mask=None, batch_mask_dir=None, animatediff_batch=False, batch_modifiers=[], batch_image_files=[]), ControlNetUnit(is_ui=True, input_mode=<InputMode.SIMPLE: 'simple'>, batch_images='', output_dir='', loopback=False, enabled=False, module='none', model='None', weight=1.0, image=None, resize_mode=<ResizeMode.INNER_FIT: 'Crop and Resize'>, low_vram=False, processor_res=-1, threshold_a=-1.0, threshold_b=-1.0, guidance_start=0.0, guidance_end=1.0, pixel_perfect=False, control_mode=<ControlMode.BALANCED: 'Balanced'>, inpaint_crop_input_image=True, hr_option=<HiResFixOption.BOTH: 'Both'>, save_detected_map=True, advanced_weighting=None, effective_region_mask=None, pulid_mode=<PuLIDMode.FIDELITY: 'Fidelity'>, ipadapter_input=None, mask=None, batch_mask_dir=None, animatediff_batch=False, batch_modifiers=[<function AnimateDiffProcess.set_p.<locals>.cn_batch_modifler at 0x000001FF9FFDFE20>], batch_image_files=[]), False, 1, 0.5, 4, 0, 0.5, 2) {}
    Traceback (most recent call last):
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\ui\generate_ui.py", line 342, in generate_ui
        return generate_ui_(*args, **kwargs)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\ui\generate_ui.py", line 323, in generate_ui_
        save_video(resultFrames, fps_out, input_video, save_video_path, seed)
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\video_tools.py", line 70, in save_video
        runFFMPEG(
      File "C:\Users\justi\Desktop\subt\stable\sd.webui\webui\extensions\sd-webui-replacer\replacer\video_tools.py", line 20, in runFFMPEG
        raise Exception(f'ffmpeg exited with code {rc}. See console for details')
    Exception: ffmpeg exited with code 1. See console for details

---
light-and-ray commented 3 months ago

I see you use animatediff with inpainting SD model, it causes OOM error and artifacts. You must set up a non-inpainting checkpoint in "Models" accordion

Also you have 12 gb vram, it's enough. The problem is only wrong model

light-and-ray commented 3 months ago

I've made mode clear console log for these cases, if ffmpeg problem was caused by no one frame was processed due to other errors: https://github.com/light-and-ray/sd-webui-replacer/commit/7ec76d2eb0fde8983337fa5e625dd2e41cfe0a45 To not confuse people

niotsujj commented 3 months ago

EDIT: Fixed, thanks!