[Bug]: Calculate the optimal GIF palette, improves quality significantly, removes banding; causes GUI to 'freeze' at 100% of generation.

MrMxyzptlk commented 8 months ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

[X] I have updated WebUI and this extension to the latest version

What happened?

If Calculate the optimal GIF palette, improves quality significantly, removes banding is enabled, this occurs; otherwise AnimateDiff Forge work perfectly.

Steps to reproduce the problem

Any prompt, any AnimateDiff settings, 'stuck' at 100% (progress bar), animation is create, GUI is 'frozen'. If I go to directory (Windows), the GIF is there, I have tried other formats; same issue.

I have changed many options in the sd-webui-forge gui and Calculate the optimal GIF palette, improves quality significantly, removes banding is the only option that causes the GUI to freeze.

What should have happened?

GIF should show in GUI after generation.

Commit where the problem happens

webui: 2.9.1 (Forge) extension: sd-webui-animatediff (Forge) c25a46d3

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

N/A defaults...

Console logs

No errors...never proceeds.
Restarting the console; Stability Matrix / Stable Diffusion Forge, does not fix the issue.

Additional information

All other tasks complete successfully.

continue-revolution commented 8 months ago

@zappityzap

zappityzap commented 8 months ago

I could not reproduce this problem. I tested on Ubuntu 22.04, and don't have access to test on Windows. Could you share an example of specific AnimateDiff settings in both the extension and the settings tab that cause the issue?

MrMxyzptlk commented 8 months ago

Screenshot 2024-03-22 102754

Screenshot 2024-03-22 102952

Settings for AnimateDiff when the issues occurs; I can 'fix' this by unchecking the 'Calculate the Optimal..."

The 'main' screen for SD is default with NO changes, other than the settings you see in the screenshot for AnimateDiff. If I ran a 512x512 txt2img as is, with a simple prompt (or any), the progress bar would stick at 100%, the GIF and PNG's would be created, but the only way to create anything else and 'unstick' the progress is to restart SD.

This is not an issue with the Calculate the Optimal... option unchecked.

Thank you.

zappityzap commented 8 months ago

Thanks. I'm still unable to reproduce here.

I noticed you're using gifsicle as well. Perhaps try turning that off?

MrMxyzptlk commented 8 months ago

I originally had the gifsicle option disabled and had the same issue. I'm not sure what else to check? Is there a debug log for AnimateDiff I can enable?

zappityzap commented 8 months ago

I looked at the code and the optimize palette option should only be used when creating GIFs. Can you confirm that you see this issue when the option is checked, but the outputs do not include GIF format?

MrMxyzptlk commented 8 months ago

This issue does NOT occur with the optimize option enabled; as long as GIF isn't checked in the settings or AnimateDiff dialog.

zappityzap commented 8 months ago

Ok, so that part of the code does two things that might be the cause. Check your venv python packages for PyAV. It should list "av" and not "pyav". My version is 11.0.0.

Then check for the piexif package. If it's there, then try turning off saving infotext under Settings > User Interface > Infotext > Write infotext to the metadata of the generated image. Remember to turn it back on after testing! If it works when not saving infotext then the issue may be with exiftool. On Windows you'll need to download exiftool manually and put the binary somewhere in your PATH.

If neither of those help, try reproducing on another computer. Or try Linux ;)

MrMxyzptlk commented 8 months ago

I have av 12.0.0 I had already manually put the exiftool binary in the Windows PATH and it seems to be functioning in Windows.

It appears that it is the infotext / exiftool issue: Here is the output without and with save infotext to image.

Without

2024-03-24 13:00:34,703 - AnimateDiff - INFO - AnimateDiff process start. 2024-03-24 13:00:34,703 - AnimateDiff - INFO - Loading motion module v3_sd15_mm.ckpt from C:\Users\Anthony\AppData\Roaming\StabilityMatrix\Packages\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\model\v3_sd15_mm.ckpt 2024-03-24 13:00:35,965 - AnimateDiff - INFO - Guessed v3_sd15_mm.ckpt architecture: MotionModuleType.AnimateDiffV3 2024-03-24 13:00:36,532 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 16 frames. INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 16 images in a total of 1 batches. 2024-03-24 13:00:36,741 - AnimateDiff - INFO - Setting DDIM alpha. 2024-03-24 13:00:36,904 - AnimateDiff - INFO - Injecting motion module v3_sd15_mm.ckpt into SD1.5 UNet. To load target model BaseModel To load target model MotionWrapper Begin to load 2 models [Memory Management] Current Free GPU Memory (MB) = 10890.419921875 [Memory Management] Model Memory (MB) = 1639.406135559082 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 8227.013786315918 [Memory Management] Current Free GPU Memory (MB) = 9237.51123046875 [Memory Management] Model Memory (MB) = 797.7752685546875 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7415.7359619140625 Moving model(s) has taken 2.47 seconds 100%|██████████| 20/20 [01:57<00:00, 5.85s/it] To load target model AutoencoderKL Begin to load 1 model [Memory Management] Current Free GPU Memory (MB) = 8384.1884765625 [Memory Management] Model Memory (MB) = 159.55708122253418 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7200.631395339966 Moving model(s) has taken 0.27 seconds 2024-03-24 13:02:42,892 - AnimateDiff - INFO - Saving output formats: GIF Installing sd-webui-animatediff GIF optimization requirement: pygifsicle gifsicle: warning: trivial adaptive palette (only 255 colors in source) 2024-03-24 13:02:49,072 - AnimateDiff - INFO - AnimateDiff process end. Total progress: 100%|██████████| 20/20 [02:03<00:00, 6.15s/it]t]

With

2024-03-24 13:03:35,035 - AnimateDiff - INFO - AnimateDiff process start. 2024-03-24 13:03:35,036 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 16 frames. INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 16 images in a total of 1 batches. 2024-03-24 13:03:35,147 - AnimateDiff - INFO - Setting DDIM alpha. 2024-03-24 13:03:35,150 - AnimateDiff - INFO - Injecting motion module v3_sd15_mm.ckpt into SD1.5 UNet. To load target model BaseModel To load target model MotionWrapper Begin to load 2 models Reuse 1 loaded models Reuse 1 loaded models [Memory Management] Current Free GPU Memory (MB) = 8215.900390625 [Memory Management] Model Memory (MB) = 0.0 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7191.900390625 [Memory Management] Current Free GPU Memory (MB) = 8215.900390625 [Memory Management] Model Memory (MB) = 0.0 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7191.900390625 Moving model(s) has taken 0.04 seconds 100%|██████████| 20/20 [01:56<00:00, 5.84s/it] 2024-03-24 13:05:37,683 - AnimateDiff - INFO - Saving output formats: GIF Installing sd-webui-animatediff GIF palette optimization requirement: PyExifTool

I already have a couple of distros installed for a while now; however I'm sicko and prefer my Windows and Mac OS's better ; ) Thank you for your help...

On Sat, Mar 23, 2024 at 11:43 AM zappityzap [notifications@github.com](mailto:notifications@github.com) wrote: Ok, so that part of the code does two things that might be the cause. Check your venv python packages for PyAV. It should list "av" and not "pyav". My version is 11.0.0.

Then check for the piexif package. If it's there, then try turning off saving infotext under Settings > User Interface > Infotext > Write infotext to the metadata of the generated image. Remember to turn it back on after testing! If it works when not saving infotext then the issue may be with exiftool. On Windows you'll need to download exiftool manually and put the binary somewhere in your PATH.

If neither of those help, try reproducing on another computer. Or try Linux ;)

MrMxyzptlk commented 8 months ago

I have av 12.0.0 I had already manually put the exiftool binary in the Windows PATH and it seems to be functioning in Windows.

It appears that it is the infotext / exiftool issue: Here is the output without and with save infotext to image.

Without

2024-03-24 13:00:34,703 - AnimateDiff - INFO - AnimateDiff process start. 2024-03-24 13:00:34,703 - AnimateDiff - INFO - Loading motion module v3_sd15_mm.ckpt from C:\Users\Anthony\AppData\Roaming\StabilityMatrix\Packages\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\model\v3_sd15_mm.ckpt 2024-03-24 13:00:35,965 - AnimateDiff - INFO - Guessed v3_sd15_mm.ckpt architecture: MotionModuleType.AnimateDiffV3 2024-03-24 13:00:36,532 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 16 frames. INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 16 images in a total of 1 batches. 2024-03-24 13:00:36,741 - AnimateDiff - INFO - Setting DDIM alpha. 2024-03-24 13:00:36,904 - AnimateDiff - INFO - Injecting motion module v3_sd15_mm.ckpt into SD1.5 UNet. To load target model BaseModel To load target model MotionWrapper Begin to load 2 models [Memory Management] Current Free GPU Memory (MB) = 10890.419921875 [Memory Management] Model Memory (MB) = 1639.406135559082 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 8227.013786315918 [Memory Management] Current Free GPU Memory (MB) = 9237.51123046875 [Memory Management] Model Memory (MB) = 797.7752685546875 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7415.7359619140625 Moving model(s) has taken 2.47 seconds 100%|██████████| 20/20 [01:57<00:00, 5.85s/it] To load target model AutoencoderKL Begin to load 1 model [Memory Management] Current Free GPU Memory (MB) = 8384.1884765625 [Memory Management] Model Memory (MB) = 159.55708122253418 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7200.631395339966 Moving model(s) has taken 0.27 seconds 2024-03-24 13:02:42,892 - AnimateDiff - INFO - Saving output formats: GIF Installing sd-webui-animatediff GIF optimization requirement: pygifsicle gifsicle: warning: trivial adaptive palette (only 255 colors in source) 2024-03-24 13:02:49,072 - AnimateDiff - INFO - AnimateDiff process end. Total progress: 100%|██████████| 20/20 [02:03<00:00, 6.15s/it]t]

With

2024-03-24 13:03:35,035 - AnimateDiff - INFO - AnimateDiff process start. 2024-03-24 13:03:35,036 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 16 frames. INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 16 images in a total of 1 batches. 2024-03-24 13:03:35,147 - AnimateDiff - INFO - Setting DDIM alpha. 2024-03-24 13:03:35,150 - AnimateDiff - INFO - Injecting motion module v3_sd15_mm.ckpt into SD1.5 UNet. To load target model BaseModel To load target model MotionWrapper Begin to load 2 models Reuse 1 loaded models Reuse 1 loaded models [Memory Management] Current Free GPU Memory (MB) = 8215.900390625 [Memory Management] Model Memory (MB) = 0.0 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7191.900390625 [Memory Management] Current Free GPU Memory (MB) = 8215.900390625 [Memory Management] Model Memory (MB) = 0.0 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 7191.900390625 Moving model(s) has taken 0.04 seconds 100%|██████████| 20/20 [01:56<00:00, 5.84s/it] 2024-03-24 13:05:37,683 - AnimateDiff - INFO - Saving output formats: GIF Installing sd-webui-animatediff GIF palette optimization requirement: PyExifTool

I already have a couple of distros installed for a while now; however I'm sicko and prefer my Windows and Mac OS's better ; ) Thank you for your help...

On Sat, Mar 23, 2024 at 11:43 AM zappityzap @.***> wrote:

Ok, so that part of the code does two things that might be the cause. Check your venv python packages for PyAV. It should list "av" and not "pyav". My version is 11.0.0.

Then check for the piexif package. If it's there, then try turning off saving infotext under Settings > User Interface > Infotext > Write infotext to the metadata of the generated image. Remember to turn it back on after testing! If it works when not saving infotext then the issue may be with exiftool. On Windows you'll need to download exiftool manually and put the binary somewhere in your PATH.

If neither of those help, try reproducing on another computer. Or try Linux ;)

— Reply to this email directly, view it on GitHub https://github.com/continue-revolution/sd-forge-animatediff/issues/10#issuecomment-2016528612, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANL7KPAZH6IYI6TAZFMZNLYZWPJNAVCNFSM6AAAAABE2MHWRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWGUZDQNRRGI . You are receiving this because you authored the thread.Message ID: @.***>

MrMxyzptlk commented 8 months ago

OK, thanks once again; I was able to fix the issue. Turns out it WASN'T in my PATH.

Not it's working.

continue-revolution / sd-forge-animatediff