Closed hoonlight closed 1 year ago
I was just about to open an issue with pretty much the same info as above:
After updating to MacOS 13.3, when using version 1.5 models, there are very infrequent failures mid-generation, (~ 1 in 20, and clustered, not evenly distributed), throwing this error:
File "/Users/jrittvo/git/Automatic1111-WebUI/modules/devices.py", line 152, in test_for_nans raise NansException(message) modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.
Running the generation again right after, with all settings left the same, works fine as often as not.
Enabling "Upcast cross attention layer to float32" seems to prevent the error from happening completely, even though these are 1.5 models.
I believe that all my 1.5 version models are ema-only fp16.
Version 2.1 models work as expected with "Upcast cross attention layer to float32" enabled, and fail to generate from the start by erroring out with it disabled, again as expected.
I think the error being reported is itself in error, and that it is interrupting what would otherwise be a successful generation. I will test later today with using the "--disable-nan-check" argument. I assume there is no downside to enabling it.
I think the error being reported is itself in error, and that it is interrupting what would otherwise be a successful generation. I will test later today with using the "--disable-nan-check" argument. I assume there is no downside to enabling it.
It seems that you are right. The tasks are proceeding without interruption after adding --disable-nan-check
. Thank you.
I want to add that I also had this issue. Even on my other version I had set up, which was pretty similar to this one.
Few events to occur:
I had better success avoiding most errors with --no-half-vae
and --disable-nan-check
@comienzo2093 @jrittvo @MDMAchine Please let me know if 20230416_experimental still gives you NansException with Upcast cross attention layer to float32
set to Automatic
.
No NansException with Automatic (or with Enabled). With Disabled it errors, as expected. Tested with tomesd enabled for both the initial and Hires fix passes, Restore faces also enabled, and including a negative prompt with Disable negative guidance minimum sigma when token merging is active
enabled.
I'm only just starting to play with this build. I spent a bit of time not getting anywhere in the other thread you solved. Thank you for that!
On this new build, a run of 4 generations with Hires fix that took 285 seconds in the previous build (and in my souped up regular Auto1111) is down to 240 seconds. That's 15% faster. Pretty impressive! I'll take a look at memory load tomorrow. No bugs at all, so far. You fixed a nasty one from the last build that crashed the app when I tried to make any selection in Maximum down sampling
on the Token Merge page.
Spoke to soon on "You fixed a nasty one from the last build that crashed the app when I tried to make any selection in Maximum down sampling on the Token Merge page." It still errors out when I hit apply settings. Looks like an easy one to fix. AssertionError: Bad value for setting token_merging_maximum_down_sampling: 1; expecting int
.
I can generate a 1536x1536 with this build on my MacBook M1Pro w/ 16GB. I bet I can do 2048x2048 if I close all my other apps and Firefox tabs. Gonna have to try the same with the official build. I've never tried anything above 1024x1024 until just now.
Spoke to soon on "You fixed a nasty one from the last build that crashed the app when I tried to make any selection in Maximum down sampling on the Token Merge page." It still errors out when I hit apply settings. Looks like an easy one to fix.
AssertionError: Bad value for setting token_merging_maximum_down_sampling: 1; expecting int
.
@jrittvo Fixed in 20230416_experimental.
I can generate a 1536x1536 with this build on my MacBook M1Pro w/ 16GB. I bet I can do 2048x2048 if I close all my other apps and Firefox tabs. Gonna have to try the same with the official build. I've never tried anything above 1024x1024 until just now.
You will probably find that 2048x2048 doesn't work, but any resolution below that should work (sometimes I generate at 2000x2000).
@comienzo2093 @jrittvo @MDMAchine Please let me know if 20230416_experimental still gives you NansException with
Upcast cross attention layer to float32
set toAutomatic
.
After testing for several hours, now I don't get that error. great!
@comienzo2093 @jrittvo @MDMAchine Please let me know if 20230416_experimental still gives you NansException with
Upcast cross attention layer to float32
set toAutomatic
.
RuntimeError: MPS backend out of memory (MPS allocated: 5.63 GB, other allocations: 13.12 GB, max allowed: 18.13 GB). Tried to allocate 768.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
Another problem is occurring. I'll open an issue after watching a bit more on this.
What size image or batch is that with? I hit that error @ 2048x2048. 1536x1536 was slow, but it did complete. Doing multiple images in a batch, as opposed to multiple batches of 1 image each, also taxes memory.
What size image or batch is that with? I hit that error @ 2048x2048. 1536x1536 was slow, but it did complete. Doing multiple images in a batch, as opposed to multiple batches of 1 image each, also taxes memory.
mbp14, fp16 checkpoint, Upcast cross attention layer to float32 : automatic DPM++SDE karras, 512x768, hires.fix, Latent, upscale by x2 batch size : 1
thank you I'll keep testing by changing the options.
Have you tried exporting PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0
as suggested in the error message? Could be a fix, or could cause a big crash. Also, for me in general, not yet with this version, there seem to be times when something "gets stuck" and generation slows down a lot. I assume its because of memory needing to swap. Shutting down Automatic1111 completely and restarting returns things to normal for me when that happens. It's not often. Once every few days.
One more thing. I've been using a 2.6 GB version of the SD-2.1-768 model (and the .yaml that pairs with it). That is half the size of the default version. I don't know if a smaller model means lower memory pressure, but it might be worth a try. It is from one of the groups that is linked to for getting the set of ControlNet models, and it is a safetensor, so I trust it. It produces the exact same results as the 5.2 GB models. https://huggingface.co/webui/stable-diffusion-2-1/tree/main
This is potentially mostly fixed in Experimental Offline Standalone Mac Installer for Stable Diffusion Web UI (unofficial) 20230727 by dfb904cb753adab19d3a632da5be401295f44f28. Please give it a try and let me know if NansException occurs as often, and which version of macOS you are currently using.
Since there have been no new reports, I'm closing this issue. If anyone still has this issue, please comment here and I'll reopen this.
Is there an existing issue for this?
What happened?
While creating the image, the process aborted with the following error.
Also, this error doesn't seem to happen every time, it seems to happen intermittently. So I'm still watching.
Steps to reproduce the problem
M1pro 14 macos 13.3 chilloutmix Fp16 model (pruned, non-ema)
512x768 SDE++ 2M Karras CFG 7 40 steps hires.fix 2x, 40 steps, swinIR_4x
What should have happened?
The image should have been created normally through the hires.fix process.
Commit where the problem happens
4b15929
What platforms do you use to access the UI ?
MacOS
What browsers do you use to access the UI ?
MS Edge
Command Line Arguments
List of extensions
Dynamic prompts
Console logs
Additional information
I've never had anything like this happen before(with automatic1111 version)