Suddenly getting ERROR:root:CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

I hope someone has some ideas I can try. It has been working fine for over a week until yesterday when it started getting these errors. My page file is set to 49gb. I was going to try the suggested downgrade to CUDA 11 and xformers but I have no python_embeded folder? I have a 4060ti 16gb My full consle log is too many characters to post here so I will post the first and last section

Full Console Log F:\ai\Fooocus-API>python .\main.py --preset realistic --gpu-device-id 0 [System ARGV] ['.\main.py', '--preset', 'realistic', '--gpu-device-id', '0'] Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Fooocus-API version: 0.3.30 Fooocus exists and URL is correct. Fooocus checkout finished for https://github.com/lllyasviel/Fooocus/commit/624f74a1ed78ea09467c856cef35aeee0af863f6. Set device to: 0 Loaded preset: F:\ai\Fooocus-API\presets\realistic.json [Fooocus-API] Task queue size: 100, queue history size: 0, webhook url: None Preload pipeline Total VRAM 16380 MB, total RAM 32702 MB Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce RTX 4060 Ti : native VAE dtype: torch.bfloat16 Using pytorch cross attention INFO: Started server process [11308] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8888/ (Press CTRL+C to quit) Refiner unloaded. model_type EPS UNet ADM Dimension 2816 Using pytorch attention in VAE Working with z of shape (1, 4, 32, 32) = 4096 dimensions. Using pytorch attention in VAE extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'} Base model loaded: F:\ai\Models\checkpoints\realisticStockPhoto_v10.safetensors Request to load LoRAs [['SDXL_FILM_PHOTOGRAPHY_STYLE_BetaV0.4.safetensors', 0.25], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [F:\ai\Models\checkpoints\realisticStockPhoto_v10.safetensors]. Loaded LoRA [F:\ai\Models\loras\SDXL_FILM_PHOTOGRAPHY_STYLE_BetaV0.4.safetensors] for UNet [F:\ai\Models\checkpoints\realisticStockPhoto_v10.safetensors] with 788 keys at weight 0.25. Loaded LoRA [F:\ai\Models\loras\SDXL_FILM_PHOTOGRAPHY_STYLE_BetaV0.4.safetensors] for CLIP [F:\ai\Models\checkpoints\realisticStockPhoto_v10.safetensors] with 264 keys at weight 0.25. Fooocus V2 Expansion: Vocab with 642 words. Fooocus Expansion engine loaded for cuda:0, use_fp16 = True. Requested to load SDXLClipModel Requested to load GPT2LMHeadModel Loading 2 new models [Fooocus Model Management] Moving model(s) has taken 1.32 seconds [Task Queue] Waiting for task finished, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c [Task Queue] Task queue start task, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c [Parameters] Adaptive CFG = 7.0 [Parameters] Sharpness = 10.0 [Parameters] ADM Scale = 1.5 : 0.8 : 0.3 [Parameters] CFG = 1.9 [Parameters] Seed = 1707547220 [Parameters] Sampler = dpmpp_2m_sde_gpu - karras [Parameters] Steps = 30 - 8 [Fooocus] Initializing ... [Fooocus] Loading models ... model_type EPS UNet ADM Dimension 2816 Using pytorch attention in VAE Working with z of shape (1, 4, 32, 32) = 4096 dimensions. Using pytorch attention in VAE [Task Queue] Already waiting for 12.7 seconds, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'} left over keys: dict_keys(['conditioner.embedders.0.logit_scale', 'conditioner.embedders.0.text_projection', 'conditioner.embedders.1.model.transformer.text_model.embeddings.position_ids']) Refiner model loaded: F:\ai\Models\checkpoints\albedobaseXL_v13.safetensors model_type EPS UNet ADM Dimension 2816 Using pytorch attention in VAE Working with z of shape (1, 4, 32, 32) = 4096 dimensions. Using pytorch attention in VAE [Task Queue] Already waiting for 25.5 seconds, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'} loaded straight to GPU Requested to load SDXL Loading 1 new model [Fooocus Model Management] Moving model(s) has taken 0.12 seconds Base model loaded: F:\ai\Models\checkpoints\formulaxlXLComfyui_v20Pruned.safetensors Request to load LoRAs [('wrong_lora.safetensors', 1.0)] for model [F:\ai\Models\checkpoints\formulaxlXLComfyui_v20Pruned.safetensors]. Loaded LoRA [F:\ai\Models\loras\wrong_lora.safetensors] for UNet [F:\ai\Models\checkpoints\formulaxlXLComfyui_v20Pruned.safetensors] with 560 keys at weight 1.0. Request to load LoRAs [('wrong_lora.safetensors', 1.0)] for model [F:\ai\Models\checkpoints\albedobaseXL_v13.safetensors]. Loaded LoRA [F:\ai\Models\loras\wrong_lora.safetensors] for UNet [F:\ai\Models\checkpoints\albedobaseXL_v13.safetensors] with 560 keys at weight 1.0. Requested to load SDXLClipModel Loading 1 new model [Fooocus Model Management] Moving model(s) has taken 0.79 seconds [Fooocus] Processing prompts ... [Fooocus] Encoding positive https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Encoding negative https://github.com/lllyasviel/Fooocus/discussions/1 ... [Parameters] Denoising Strength = 1.0 [Parameters] Initial Latent shape: Image Space (1152, 896) Preparation time: 32.49 seconds [Sampler] refiner_swap_method = joint [Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828 Requested to load SDXL Loading 1 new model unload clone 2 [Fooocus Model Management] Moving model(s) has taken 3.84 seconds 7%|███████████▍ | 2/30 [00:01<00:21, 1.30it/s][Task Queue] Already waiting for 38.4 seconds, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c 27%|█████████████████████████████████████████████▊ | 8/30 [00:04<00:11, 1.94it/s]Requested to load SDXL Loading 1 new model [Fooocus Model Management] Moving model(s) has taken 1.92 seconds Refiner Swapped 80%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 24/30 [00:14<00:03, 2.00it/s][Task Queue] Already waiting for 51.1 seconds, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:17<00:00, 1.71it/s] Requested to load AutoencoderKL Loading 1 new model [Fooocus Model Management] Moving model(s) has taken 0.77 seconds Image generated with private log at: F:\ai\Images\TempFiles\2024-02-10\log.html Generating and saving time: 24.05 seconds [Task Queue] Finish task, job_id=97bf2e56-f680-471f-9402-c7c1631a8f2c Requested to load SDXLClipModel Requested to load GPT2LMHeadModel Loading 2 new models INFO: 127.0.0.1:53062 - "POST /v1/generation/text-to-image HTTP/1.1" 200 OK [INFO] Mixing image prompt and vary upscale is set to True [Task Queue] Waiting for task finished, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b [Fooocus Model Management] Moving model(s) has taken 0.72 seconds [Task Queue] Task queue start task, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b [Parameters] Adaptive CFG = 7.0 [Parameters] Sharpness = 10.0 [Parameters] ADM Scale = 1.5 : 0.8 : 0.3 [Parameters] CFG = 1.9 [Parameters] Seed = 1707547220 [Fooocus] Downloading upscale models ... [Fooocus] Downloading control models ... [Fooocus] Loading control models ... [Parameters] Sampler = dpmpp_2m_sde_gpu - karras [Parameters] Steps = 18 - 5 [Fooocus] Initializing ... [Fooocus] Loading models ... [Fooocus] Processing prompts ... [Fooocus] Encoding positive https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Encoding negative https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Image processing ... [Fooocus] Upscaling image from (1152, 896) ... Upscaling image with shape (1152, 896, 3) ... [Task Queue] Already waiting for 12.7 seconds, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b [Task Queue] Already waiting for 25.5 seconds, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b Image upscaled. [Fooocus] VAE encoding ... [Fooocus Model Management] Moving model(s) has taken 0.28 seconds [Task Queue] Already waiting for 38.7 seconds, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b Final resolution is (1536, 1200). [Parameters] Denoising Strength = 0.382 [Parameters] Initial Latent shape: torch.Size([1, 4, 192, 150]) Preparation time: 40.47 seconds [Sampler] refiner_swap_method = joint [Sampler] sigma_min = 0.0291671771556139, sigma_max = 0.5689725279808044 28%|███████████████████████████████████████████████▊ | 5/18 [00:04<00:12, 1.02it/s]Refiner Swapped 56%|███████████████████████████████████████████████████████████████████████████████████████████████ | 10/18 [00:09<00:07, 1.02it/s][Task Queue] Already waiting for 51.5 seconds, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:17<00:00, 1.02it/s] [Task Queue] Already waiting for 64.3 seconds, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b Image generated with private log at: F:\ai\Images\TempFiles\2024-02-10\log.html Generating and saving time: 24.78 seconds [Task Queue] Finish task, job_id=03318bf3-ec4c-473c-a42a-78152038ea3b

INFO: 127.0.0.1:53439 - "POST /v2/generation/image-upscale-vary HTTP/1.1" 200 OK [Task Queue] Waiting for task finished, job_id=1448eb78-0b8e-44e1-bfbc-5b408f2e24eb [Task Queue] Task queue start task, job_id=1448eb78-0b8e-44e1-bfbc-5b408f2e24eb [Parameters] Adaptive CFG = 7.0 [Parameters] Sharpness = 10.0 [Parameters] ADM Scale = 1.5 : 0.8 : 0.3 [Parameters] CFG = 1.9 [Parameters] Seed = 1707550427 [Parameters] Sampler = dpmpp_2m_sde_gpu - karras [Parameters] Steps = 30 - 8 [Fooocus] Initializing ... [Fooocus] Loading models ... [Fooocus] Processing prompts ... [Fooocus] Encoding positive https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Encoding negative https://github.com/lllyasviel/Fooocus/discussions/1 ... [Parameters] Denoising Strength = 1.0 [Parameters] Initial Latent shape: Image Space (1152, 896) Preparation time: 0.75 seconds [Sampler] refiner_swap_method = joint [Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828 27%|█████████████████████████████████████████████▊ | 8/30 [00:04<00:11, 2.00it/s]Refiner Swapped 77%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 23/30 [00:11<00:03, 2.00it/s][Task Queue] Already waiting for 12.8 seconds, job_id=1448eb78-0b8e-44e1-bfbc-5b408f2e24eb 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:14<00:00, 2.00it/s] [Fooocus Model Management] Moving model(s) has taken 0.87 seconds Image generated with private log at: F:\ai\Images\TempFiles\2024-02-10\log.html Generating and saving time: 16.97 seconds [Task Queue] Finish task, job_id=1448eb78-0b8e-44e1-bfbc-5b408f2e24eb Requested to load SDXLClipModel Requested to load GPT2LMHeadModel Loading 2 new models INFO: 127.0.0.1:53441 - "POST /v1/generation/text-to-image HTTP/1.1" 200 OK [INFO] Mixing image prompt and vary upscale is set to True [Task Queue] Waiting for task finished, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a [Fooocus Model Management] Moving model(s) has taken 0.74 seconds [Task Queue] Task queue start task, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a [Parameters] Adaptive CFG = 7.0 [Parameters] Sharpness = 10.0 [Parameters] ADM Scale = 1.5 : 0.8 : 0.3 [Parameters] CFG = 1.9 [Parameters] Seed = 1707550427 [Fooocus] Downloading upscale models ... [Fooocus] Downloading control models ... [Fooocus] Loading control models ... [Parameters] Sampler = dpmpp_2m_sde_gpu - karras [Parameters] Steps = 18 - 5 [Fooocus] Initializing ... [Fooocus] Loading models ... [Fooocus] Processing prompts ... [Fooocus] Encoding positive https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Encoding negative https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Image processing ... [Fooocus] Upscaling image from (1152, 896) ... Upscaling image with shape (1152, 896, 3) ... [Task Queue] Already waiting for 12.7 seconds, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a [Task Queue] Already waiting for 25.5 seconds, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a [Task Queue] Already waiting for 38.3 seconds, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a Image upscaled. [Fooocus] VAE encoding ... [Fooocus Model Management] Moving model(s) has taken 0.31 seconds Final resolution is (1536, 1200). [Parameters] Denoising Strength = 0.382 [Parameters] Initial Latent shape: torch.Size([1, 4, 192, 150]) Preparation time: 42.46 seconds [Sampler] refiner_swap_method = joint [Sampler] sigma_min = 0.0291671771556139, sigma_max = 0.5689725279808044 28%|███████████████████████████████████████████████▊ | 5/18 [00:04<00:12, 1.03it/s]Refiner Swapped 44%|████████████████████████████████████████████████████████████████████████████▍ | 8/18 [00:07<00:09, 1.02it/s][Task Queue] Already waiting for 51.5 seconds, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:17<00:00, 1.03it/s] [Task Queue] Already waiting for 64.2 seconds, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a Image generated with private log at: F:\ai\Images\TempFiles\2024-02-10\log.html Generating and saving time: 24.13 seconds [Task Queue] Finish task, job_id=6006ca6d-d5cc-4ae6-ba7c-6557fa956e7a INFO: 127.0.0.1:53443 - "POST /v2/generation/image-upscale-vary HTTP/1.1" 200 OK [Task Queue] Waiting for task finished, job_id=71e2ba79-a373-435c-b820-5909297e0edc [Task Queue] Task queue start task, job_id=71e2ba79-a373-435c-b820-5909297e0edc [Parameters] Adaptive CFG = 7.0 [Parameters] Sharpness = 10.0 [Parameters] ADM Scale = 1.5 : 0.8 : 0.3 [Parameters] CFG = 1.9 [Parameters] Seed = 1707550514 [Parameters] Sampler = dpmpp_2m_sde_gpu - karras [Parameters] Steps = 30 - 8 [Fooocus] Initializing ... [Fooocus] Loading models ... [Fooocus] Processing prompts ... [Fooocus] Encoding positive https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Encoding negative https://github.com/lllyasviel/Fooocus/discussions/1 ... [Parameters] Denoising Strength = 1.0 [Parameters] Initial Latent shape: Image Space (1152, 896) Preparation time: 0.72 seconds [Sampler] refiner_swap_method = joint [Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828 27%|█████████████████████████████████████████████▊ | 8/30 [00:03<00:10, 2.00it/s]Refiner Swapped 77%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 23/30 [00:11<00:03, 2.00it/s][Task Queue] Already waiting for 12.7 seconds, job_id=71e2ba79-a373-435c-b820-5909297e0edc 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:14<00:00, 2.00it/s] [Fooocus Model Management] Moving model(s) has taken 0.95 seconds Image generated with private log at: F:\ai\Images\TempFiles\2024-02-10\log.html Generating and saving time: 17.03 seconds [Task Queue] Finish task, job_id=71e2ba79-a373-435c-b820-5909297e0edc Requested to load SDXLClipModel Requested to load GPT2LMHeadModel Loading 2 new models INFO: 127.0.0.1:53447 - "POST /v1/generation/text-to-image HTTP/1.1" 200 OK [INFO] Mixing image prompt and vary upscale is set to True [Task Queue] Waiting for task finished, job_id=6e416888-81d9-45d8-9b42-502583ff6231 [Fooocus Model Management] Moving model(s) has taken 0.72 seconds [Task Queue] Task queue start task, job_id=6e416888-81d9-45d8-9b42-502583ff6231 [Parameters] Adaptive CFG = 7.0 [Parameters] Sharpness = 10.0 [Parameters] ADM Scale = 1.5 : 0.8 : 0.3 [Parameters] CFG = 1.9 [Parameters] Seed = 1707550514 [Fooocus] Downloading upscale models ... [Fooocus] Downloading control models ... [Fooocus] Loading control models ... [Parameters] Sampler = dpmpp_2m_sde_gpu - karras [Parameters] Steps = 18 - 5 [Fooocus] Initializing ... [Fooocus] Loading models ... [Fooocus] Processing prompts ... [Fooocus] Encoding positive https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Encoding negative https://github.com/lllyasviel/Fooocus/discussions/1 ... [Fooocus] Image processing ... [Fooocus] Upscaling image from (1152, 896) ... Upscaling image with shape (1152, 896, 3) ... Worker error: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

ERROR:root:CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. Traceback (most recent call last): File "F:\ai\Fooocus-API\fooocusapi\worker.py", line 513, in process_generate uov_input_image = perform_upscale(uov_input_image) File "F:\ai\Fooocus-API\repositories\Fooocus\modules\upscaler.py", line 31, in perform_upscale img = opImageUpscaleWithModel.upscale(model, img)[0] File "F:\ai\Fooocus-API\repositories\Fooocus\ldm_patched\contrib\external_upscale_model.py", line 54, in upscale s = ldm_patched.modules.utils.tiled_scale(in_img, lambda a: upscale_model(a), tile_x=tile, tile_y=tile, overlap=overlap, upscale_amount=upscale_model.scale, pbar=pbar) File "C:\Users\Teedo\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "F:\ai\Fooocus-API\repositories\Fooocus\ldm_patched\modules\utils.py", line 418, in tiled_scale ps = function(s_in).to(output_device) RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Task Queue] Finish task with error, job_id=6e416888-81d9-45d8-9b42-502583ff6231 INFO: 127.0.0.1:53453 - "POST /v2/generation/image-upscale-vary HTTP/1.1" 200 OK

mrhan1993 / Fooocus-API

Suddenly getting ERROR:root:CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. #213