[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui
Have you read FAQ on README?
[X] I have updated WebUI and this extension to the latest version
What happened?
Upon trying to generate a moving image, based on an mp4 using controlnet. The application will crash because neither FFMPEG or OpenCV can process the file due to a wrong path definition (generated through the code). All follow up steps in controlnet will then fail.
Steps to reproduce the problem
Go to Animatediff and enable it.
Load an mp4 as source
Enable controlnet (i usually use depth_zoe & openpose, but this problem happens before it reaches controlnet. Just need to enable it to trigger the frame extraction)
Press generate. The programn will crash upon the frame extraction from your mp4.
What should have happened?
The programn should extract the frames from the provided video input and then feed them into controlnet for the preprocessors to process.
All other extensions (except built in) disabled to isolate problem.
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
--xformers
Console logs
2024-03-05 10:46:57,540 - AnimateDiff - ERROR - [AnimateDiff] Error extracting frames via ffmpeg: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\\Users\\redacted\\AppData\\Local\\Temp\\gradio\\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\\hipsway.mp4-3e5b58ae', fall back to OpenCV.
2024-03-05 10:46:57,540 - AnimateDiff - INFO - Attempting to extract frames via OpenCV from C:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4 to GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3e5b58ae
*** Error running before_process: E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
Traceback (most recent call last):
File "E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 102, in extract_frames_from_video
ffmpeg_extract_frames(params.video_source, params.video_path)
File "E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 73, in ffmpeg_extract_frames
tmp_frame_dir.mkdir(parents=True, exist_ok=True)
File "C:\Program Files\Python310\lib\pathlib.py", line 1175, in mkdir
self._accessor.mkdir(self, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\\Users\\redacted\\AppData\\Local\\Temp\\gradio\\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\\hipsway.mp4-3e5b58ae'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\AI\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
script.before_process(p, *script_args)
File "E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 64, in before_process
params.set_p(p)
File "E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 169, in set_p
extract_frames_from_video(self)
File "E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 105, in extract_frames_from_video
cv2_extract_frames(params.video_source, params.video_path)
File "E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 84, in cv2_extract_frames
tmp_frame_dir.mkdir(parents=True, exist_ok=True)
File "C:\Program Files\Python310\lib\pathlib.py", line 1175, in mkdir
self._accessor.mkdir(self, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\\Users\\redacted\\AppData\\Local\\Temp\\gradio\\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\\hipsway.mp4-3e5b58ae'
---
2024-03-05 10:46:58,305 - ControlNet - INFO - unit_separate = False, style_align = False
Additional information
Have tested with both a new input file, as well as an old one that used to work (both mp4).
I modified the util.py file afterwards. To see what the path values are before being used, and how they get mutated. Figuring it may be a simply forgot a space problem. But i couldn't see the exact fix.
The following modification:
def extract_frames_from_video(params):
assert params.video_source, "You need to specify cond hint for ControlNet."
logger.info(f"Base data path {data_path}")
params.video_path = shared.opts.data.get(
"animatediff_frame_extract_path",
f"{data_path}/tmp/animatediff-frames")
logger.info(f"Frame extraction path after get {params.video_path}")
params.video_path += f"{params.video_source}-{generate_random_hash()}"
logger.info(f"Frame extraction path after += {params.video_path}")
try:
ffmpeg_extract_frames(params.video_source, params.video_path)
except Exception as e:
logger.error(f"[AnimateDiff] Error extracting frames via ffmpeg: {e}, fall back to OpenCV.")
cv2_extract_frames(params.video_source, params.video_path)
Gives the following output:
2024-03-05 11:12:05,336 - AnimateDiff - INFO - Injection finished.
2024-03-05 11:12:05,337 - AnimateDiff - INFO - Base data path E:\AI\stable-diffusion-webui
2024-03-05 11:12:05,337 - AnimateDiff - INFO - Frame extraction path after get GIF
2024-03-05 11:12:05,338 - AnimateDiff - INFO - Frame extraction path after += GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3a587949
2024-03-05 11:12:05,338 - AnimateDiff - ERROR - [AnimateDiff] Error extracting frames via ffmpeg: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3a587949', fall back to OpenCV.
2024-03-05 11:12:05,339 - AnimateDiff - INFO - Attempting to extract frames via OpenCV from C:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4 to GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3a587949
*** Error running before_process: E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
Is there an existing issue for this?
Have you read FAQ on README?
What happened?
Upon trying to generate a moving image, based on an mp4 using controlnet. The application will crash because neither FFMPEG or OpenCV can process the file due to a wrong path definition (generated through the code). All follow up steps in controlnet will then fail.
Steps to reproduce the problem
What should have happened?
The programn should extract the frames from the provided video input and then feed them into controlnet for the preprocessors to process.
Commit where the problem happens
webui: https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/bef51aed032c0aaa5cfd80445bc4cf0d85b408b5 extensions: Animatediff (https://github.com/continue-revolution/sd-webui-animatediff/commit/ba19e78e11006b4b816c26932c31ce9b877fe376) Controlnet (https://github.com/Mikubill/sd-webui-controlnet/commit/489a399ef06cb94cc8c580ecbf91c5fa1bcce111)
All other extensions (except built in) disabled to isolate problem.
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
Console logs
Additional information
Have tested with both a new input file, as well as an old one that used to work (both mp4).
I modified the util.py file afterwards. To see what the path values are before being used, and how they get mutated. Figuring it may be a simply forgot a space problem. But i couldn't see the exact fix.
The following modification:
def extract_frames_from_video(params): assert params.video_source, "You need to specify cond hint for ControlNet." logger.info(f"Base data path {data_path}") params.video_path = shared.opts.data.get( "animatediff_frame_extract_path", f"{data_path}/tmp/animatediff-frames") logger.info(f"Frame extraction path after get {params.video_path}") params.video_path += f"{params.video_source}-{generate_random_hash()}" logger.info(f"Frame extraction path after += {params.video_path}") try: ffmpeg_extract_frames(params.video_source, params.video_path) except Exception as e: logger.error(f"[AnimateDiff] Error extracting frames via ffmpeg: {e}, fall back to OpenCV.") cv2_extract_frames(params.video_source, params.video_path)
Gives the following output:
2024-03-05 11:12:05,336 - AnimateDiff - INFO - Injection finished. 2024-03-05 11:12:05,337 - AnimateDiff - INFO - Base data path E:\AI\stable-diffusion-webui 2024-03-05 11:12:05,337 - AnimateDiff - INFO - Frame extraction path after get GIF 2024-03-05 11:12:05,338 - AnimateDiff - INFO - Frame extraction path after += GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3a587949 2024-03-05 11:12:05,338 - AnimateDiff - ERROR - [AnimateDiff] Error extracting frames via ffmpeg: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3a587949', fall back to OpenCV. 2024-03-05 11:12:05,339 - AnimateDiff - INFO - Attempting to extract frames via OpenCV from C:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4 to GIFC:\Users\redacted\AppData\Local\Temp\gradio\cd80e57be5c03bd824ddfd17c493df8a72e5a4d4\hipsway.mp4-3a587949 *** Error running before_process: E:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py