Closed gateway closed 1 year ago
JamesDooley
commented
1 year ago I want to add to this that my environment was working fine until the most recent pull. Now I receive the same error as above, so this appears to be an issue with one of the most recent changes.
Refreshing the environment with the install script does not resolve the issue, nor does a reboot.
addianto
commented
1 year ago Happened to me as well on latest commit (8cbb50c204b7ae24f805574aa59ede1ed2156318) only if I tried generating an image with a long prompt such as list of keywords obtained from DeepDanbooru-based interrogator.
Barafu
commented
1 year ago I have the same symptoms, but different log!
The interesting part is:
> Ksampler using model noise schedule (steps >= 30)
>> Sampling with k_lms starting at step 0 of 50 (50 new sampling steps)
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [408,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.The offending prompt is: High detail RAW color sensual photo of ( Pare 25yo woman) with perfect face and perfect bright eyes, pale skin and freckles, large eyes, (wearing robe) (cone hair bun) realistic, highly detailed, harsh lighting, cinematic lighting, art by artgerm and greg rutkowski and alphonse mucha, (performing magic with fireflies, embers swirling around her) hasselblad, 45 degree, hard light, gigapixel. Cutting it approximately by half fixes the issue. Tell me if you need a precise value.
Win 10, GTX3070Ti, 8GB VRAM. Everything updated.
gombolduc
commented
1 year ago I get the same error
System config requested System config requested Image generation requested: {'prompt': 'A figure stands in a surreal landscape of rolling hills, strange trees and a pink sea. ', 'iterations': 1, 'steps': 50, 'cfg_scale': 7.5, 'threshold': 0, 'perlin': 0, 'height': 512, 'width': 512, 'sampler_name': 'k_euler_a', 'seed': 1414890566, 'progress_images': False, 'progress_latents': True, 'save_intermediates': 5, 'generation_mode': 'txt2img', 'init_mask': '...', 'seamless': False, 'hires_fix': True, 'variation_amount': 0} ESRGAN parameters: False Facetool parameters: {'type': 'gfpgan', 'strength': 0.8} {'prompt': 'A figure stands in a surreal landscape of rolling hills, strange trees and a pink sea. ', 'iterations': 1, 'steps': 50, 'cfg_scale': 7.5, 'threshold': 0, 'perlin': 0, 'height': 512, 'width': 512, 'sampler_name': 'k_euler_a', 'seed': 1414890566, 'progress_images': False, 'progress_latents': True, 'save_intermediates': 5, 'generation_mode': 'txt2img', 'init_mask': '', 'seamless': False, 'hires_fix': True, 'variation_amount': 0} Traceback (most recent call last): File "c:\invoke\invokeai.venv\lib\site-packages\ldm\generate.py", line 462, in prompt2image uc, c, extra_conditioning_info = get_uc_and_c_and_ec( File "c:\invoke\invokeai.venv\lib\site-packages\ldm\invoke\conditioning.py", line 24, in get_uc_and_c_and_ec conditioning = _get_conditioning_for_prompt(prompt, negative_prompt, model, log_tokens) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\invoke\conditioning.py", line 100, in _get_conditioning_forprompt conditioning, = _get_embeddings_and_tokens_for_prompt(model, File "c:\invoke\invokeai.venv\lib\site-packages\ldm\invoke\conditioning.py", line 224, in _get_embeddings_and_tokens_for_prompt embeddings, tokens = model.get_learned_conditioning([fragments], return_tokens=True, fragment_weights=[weights]) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\models\diffusion\ddpm.py", line 829, in get_learned_conditioning c = self.cond_stage_model.encode( File "c:\invoke\invokeai.venv\lib\site-packages\ldm\modules\encoders\modules.py", line 461, in encode return self(text, *kwargs) File "c:\invoke\invokeai.venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(input, **kwargs) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\modules\encoders\modules.py", line 506, in forward tokens, per_token_weights = self.get_tokens_and_weights(fragments, weights) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\modules\encoders\modules.py", line 634, in get_tokens_and_weights all_tokens_tensor = torch.tensor(all_tokens, dtype=torch.long).to(self.device) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Could not generate image. Usage stats: 0 image(s) generated in 0.03s Max VRAM used for this generation: 2.17G. Current VRAM utilization: 2.17G Max VRAM used since script start: 3.29G
saffmila
commented
1 year ago Same problem.. Works ok when i am using short prompt, but long prompt causes problem.. Update or restart doesn`t help
gateway
commented
1 year ago anyone have a fix? Any devs check this out?
I can also verify it works with shorter prompts..
Malbathla
commented
1 year ago Having the same problem. For me however it appears to occur when the prompt has been truncated more than 1 token (ie. truncate 1 - generates ok, truncate 2 or more and generates an error.)
Example Steps: 25, CFG: 7.5, Size: 1024x1024, Sampler: k_lms
FINE: No Truncated Prompt an (ultra-realistic)+ (full length portrait)+ of a futuristic (mechanical)+ female robot, looks like a celebrity, cyberpunk style, (frontal)+, (standing)++, (centered)+, (stretching)++, (full shot)+, full length, feet, elegant, (highly detailed)++, intricate, tone mapped, (Photo)++, (Realistic)+++, Canon50, (HD)++, (detailed)+, smooth, sharp focus, colorful, blade runner style city background, attractive
FINE: truncates 1 prompt token an (ultra-realistic)+ (full length portrait)+ of a futuristic (mechanical)+ female robot, looks like a celebrity, cyberpunk style, (frontal)+, (standing)++, (centered)+, (stretching)++, (full shot)+, full length, feet, elegant, (highly detailed)++, intricate, tone mapped, (Photo)++, (Realistic)+++, Canon50, (HD)++, (detailed)+, smooth, sharp focus, colorful, blade runner style city background, attractive, closeup
ERROR: truncates 2 prompt tokens an (ultra-realistic)+ (full length portrait)+ of a futuristic (mechanical)+ female robot, looks like a celebrity, cyberpunk style, (frontal)+, (standing)++, (centered)+, (stretching)++, (full shot)+, full length, feet, elegant, (highly detailed)++, intricate, tone mapped, (Photo)++, (Realistic)+++, Canon50, (HD)++, (detailed)+, smooth, sharp focus, colorful, blade runner style city background, attractive, closeup face
This required completely closing the application and restarting to continue with InvokeAI
JamesDooley
commented
1 year ago Decided to take a closer look at this since there seems to be no traction in getting this fixed.
The original issue appears to be caused by commit 786b8878d6c6a7c8db241e340c4551e71c1d6f15. Doing a git reset --hard on the commit before this resolves all issues.
This commit does generate a different error that is related to a fix in 12a8d7fc1494c24ba42fafe6c8e22249961d3bc9 once that fix is applied the current error is presented.
The 786b8878d6c6a7c8db241e340c4551e71c1d6f15 commit shows a lot of different files being updated, not sure yet where the actual issue is. Looking at #1866 I do see other issue the commit caused already being fixed.
JamesDooley
commented
1 year ago cross_attention_maps_saving.py seems to be where the code dies off for me. Specifically when calling add_attention_maps.
When I use the first prompt from malbathla, I get a tokens_id range of (1,73) and everything works properly. When I use the third prompt I get tokens_id range of (1,78)
The code does mention that the range should typically be 77, so I am guessing there should be some sort of split there if there are more tokens in the prompt.
When the end range goes above 77 the maps = maps[:,:, self.token_ids] call causes the index out of bounds error.
As a work around, to just get stuff working, on the latest commit. If you add a return directly after the setup_attention_map_saving callback definition in shared_invokeai_diffusion.py everything seems to work as expected: ~ line 56:
def setup_attention_map_saving(self, saver: AttentionMapSaver):
def callback(slice, dim, offset, slice_size, key):
returnCan confirm that this work around works well and I can now generate images regardless of prompt size - nice. Given that this is a system breaking problem I am surprised it hasn't been addessed and fixed - thank goodness for community I guess.
hipsterusername
commented
1 year ago Hey all - Monitoring issue. Unable to recreate it on my side, but recognize this bit is causing issues somewhere.
For now, my recommendation would be to use prompts that fit within the token limit, and take the silver lining view that this helps you identify when your prompting is going to waste 💁♂️
Thanks for the digging @JamesDooley
hipsterusername
commented
1 year ago AH HA.
Can folks confirm that this is only happening on the unified canvas? I.e., it doesn't have issues on the Txt2Img tab?
JamesDooley
commented
1 year ago When I run text to image, I get the same issue, but I do see an additional message about the prompt being too long and that it was truncated. It still errors out with index out of bounds.
>> Setting Sampler to k_euler_a
>> Prompt is 2 token(s) too long and has been truncated
Generating: 0%| | 0/12 [00:00<?, ?it/s]>> Ksampler using model noise schedule (steps >= 30)
>> Sampling with k_euler_ancestral starting at step 0 of 40 (40 new sampling steps)
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [27,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [108,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.All of my previous tests were against the unified canvas.
hipsterusername
commented
1 year ago Got it - Assuming this is without the fix you posted above?
JamesDooley
commented
1 year ago Correct, I commented out the early return so it was running the original code.
hipsterusername
commented
1 year ago Ok - Thanks for the additional details.
gateway
commented
1 year ago any pushed fix for this yet? Asking us to do smaller prompts is not a solution, and errors should always bee handled gracefully .. I know their is a lot going on with this amazing product but just curious where this currently stands as fixed
AskAlice
commented
1 year ago @lstein can you cut a release for this? Getting this at the moment
ogzander
commented
1 year ago @lstein can you cut a release for this? Getting this at the moment The automatic installer doesn't seem to be updated yet, but you can work around it by doing this: Go to: https://github.com/invoke-ai/InvokeAI/tree/main/ldm/invoke Copy the file or the content of "conditioning.py" and either replace or copy and paste the content into your conditioning.py located in your invokeai folder > venv\Lib\site-packages\ldm\invoke
Is there an existing issue for this?
OS
Windows
GPU
cuda
VRAM
24
What happened?
after a long long manual install since the binary one just wont work for some reason I was able to get all the required packages set up from the env, base and cuda requirements. I can launch the app and load it from my local lan but anytime I try to create an image it bails on RuntimeError: CUDA error: device-side assert triggered.
conda list