search

Haidra-Org / AI-Horde-Worker

This repo turns your PC into a AI Horde worker node
GNU Affero General Public License v3.0
242 stars 67 forks source link

stable_diffusion_inpainting with no alpha channel puts worker into maintenance? #69

Closed jug-dev closed 1 year ago

jug-dev commented 1 year ago

Log provided by TroyBORG:


INFO       | 2023-03-03 00:59:36.314204 | worker.jobs.framework:submit_job:137 - Submitted job with id 4f53b779-c4a0-471a-86f9-8e93e6ccb56d and contributed for 13.14
DEBUG      | 2023-03-03 00:59:37.927499 | nataili.stable_diffusion.compvis:generate:672 - [Low VRAM] after sampling - model.device = cpu, model.cond_stage_model.device = cpu, model.first_stage_model.device = cuda:0
DEBUG      | 2023-03-03 00:59:37.927499 | nataili.stable_diffusion.compvis:generate:873 - [Low VRAM] decode - model.first_stage_model.device = cuda:0
DEBUG      | 2023-03-03 00:59:38.071926 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.400418 (node: kai:7002:4.1.0)
INFO       | 2023-03-03 00:59:38.320071 | worker.jobs.stable_diffusion:start_job:320 - Generation finished successfully.
DEBUG      | 2023-03-03 00:59:38.321049 | worker.jobs.stable_diffusion:start_job:361 - post-processing done...
DEBUG      | 2023-03-03 00:59:38.323001 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:38.325929 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.703s (Total Completed: 9998)
DEBUG      | 2023-03-03 00:59:38.346436 | worker.workers.framework:start_job:116 - New job processing
DEBUG      | 2023-03-03 00:59:38.346436 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG      | 2023-03-03 00:59:38.347415 | worker.jobs.stable_diffusion:start_job:148 - ['stable_diffusion_inpainting', 'stable diffusion 1', 'img2img', 'txt2img']
WARNING    | 2023-03-03 00:59:38.347415 | worker.jobs.stable_diffusion:start_job:157 - Model stable_diffusion_inpainting chosen for txt2img or img2img gen, switching to Liberty instead.
DEBUG      | 2023-03-03 00:59:38.349017 | worker.jobs.stable_diffusion:start_job:200 - txt2img (Liberty) request with id 959f1d4e-a91d-4e1c-a057-7daa7a0f86d1 picked up. Initiating work...
INFO       | 2023-03-03 00:59:38.349017 | worker.jobs.stable_diffusion:start_job:312 - Starting generation: Liberty @ 512x512 for 30 steps. Prompt length is 175 characters And it appears to contain 0 weights
DEBUG      | 2023-03-03 00:59:38.751956 | nataili.stable_diffusion.compvis:generate:584 - model.device = cuda:0, model.cond_stage_model.device = cuda:0, model.first_stage_model.device = cuda:0
DEBUG      | 2023-03-03 00:59:38.751956 | nataili.stable_diffusion.compvis:generate:589 - Iteration: 1/1
DEBUG      | 2023-03-03 00:59:38.764650 | ldm.modules.encoders.modules:forward:301 - Token count for prompt = 49
DEBUG      | 2023-03-03 00:59:38.876250 | nataili.stable_diffusion.compvis:generate:641 - [Low VRAM] before sampling - model.device = cuda:0, model.cond_stage_model.device = cpu, model.first_stage_model.device = cpu
DEBUG      | 2023-03-03 00:59:38.877226 | ldm.models.diffusion.kdiffusion:sample:143 - model.device: cuda:0
DEBUG      | 2023-03-03 00:59:39.081555 | worker.jobs.stable_diffusion:prepare_submit_payload:376 - R2 Upload response: <Response [200]>
DEBUG      | 2023-03-03 00:59:39.082071 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:40.025794 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.625835 (node: goblin:7001:4.1.0)
DEBUG      | 2023-03-03 00:59:40.712635 | worker.jobs.framework:submit_job:113 - Upload completed in 1.628548
INFO       | 2023-03-03 00:59:40.713156 | worker.jobs.framework:submit_job:137 - Submitted job with id b4542659-b5a4-4a5b-a7c9-b8d2da215457 and contributed for 5.54
DEBUG      | 2023-03-03 00:59:40.847607 | nataili.stable_diffusion.compvis:generate:672 - [Low VRAM] after sampling - model.device = cpu, model.cond_stage_model.device = cpu, model.first_stage_model.device = cuda:0
DEBUG      | 2023-03-03 00:59:40.847607 | nataili.stable_diffusion.compvis:generate:873 - [Low VRAM] decode - model.first_stage_model.device = cuda:0
INFO       | 2023-03-03 00:59:41.233947 | worker.jobs.stable_diffusion:start_job:320 - Generation finished successfully.
DEBUG      | 2023-03-03 00:59:41.234925 | worker.jobs.stable_diffusion:start_job:361 - post-processing done...
DEBUG      | 2023-03-03 00:59:41.235901 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:41.239805 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.891s (Total Completed: 9999)
DEBUG      | 2023-03-03 00:59:41.259834 | worker.workers.framework:start_job:116 - New job processing
DEBUG      | 2023-03-03 00:59:41.259834 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG      | 2023-03-03 00:59:41.259834 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id bf3ddfa9-f62d-4e22-90c1-e611c0da47bf picked up. Initiating work...
WARNING    | 2023-03-03 00:59:41.260354 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG      | 2023-03-03 00:59:41.261397 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:41.261902 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:42.038881 | worker.jobs.stable_diffusion:prepare_submit_payload:376 - R2 Upload response: <Response [200]>
DEBUG      | 2023-03-03 00:59:42.038881 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:42.460393 | worker.jobs.framework:submit_job:113 - Upload completed in 1.196615
INFO       | 2023-03-03 00:59:42.460393 | worker.jobs.framework:submit_job:137 - Submitted job with id bf3ddfa9-f62d-4e22-90c1-e611c0da47bf and contributed for 0.0
DEBUG      | 2023-03-03 00:59:42.963758 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.680121 (node: thrr:7001:4.1.0)
DEBUG      | 2023-03-03 00:59:43.211049 | worker.jobs.framework:submit_job:113 - Upload completed in 1.170281
INFO       | 2023-03-03 00:59:43.211049 | worker.jobs.framework:submit_job:137 - Submitted job with id 959f1d4e-a91d-4e1c-a057-7daa7a0f86d1 and contributed for 6.5
DEBUG      | 2023-03-03 00:59:43.509148 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.250s (Total Completed: 10000)
DEBUG      | 2023-03-03 00:59:43.530443 | worker.workers.framework:start_job:116 - New job processing
DEBUG      | 2023-03-03 00:59:43.530443 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG      | 2023-03-03 00:59:43.530443 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id 11af04bc-8d01-4fc8-a380-b51c6e91a72a picked up. Initiating work...
WARNING    | 2023-03-03 00:59:43.531421 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG      | 2023-03-03 00:59:43.532396 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:43.532396 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:44.441255 | worker.jobs.framework:submit_job:113 - Upload completed in 0.906672
INFO       | 2023-03-03 00:59:44.441781 | worker.jobs.framework:submit_job:137 - Submitted job with id 11af04bc-8d01-4fc8-a380-b51c6e91a72a and contributed for 0.0
DEBUG      | 2023-03-03 00:59:45.847746 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.294338 (node: kai:7002:4.1.0)
DEBUG      | 2023-03-03 00:59:46.399698 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.859s (Total Completed: 10001)
DEBUG      | 2023-03-03 00:59:46.419986 | worker.workers.framework:start_job:116 - New job processing
DEBUG      | 2023-03-03 00:59:46.419986 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG      | 2023-03-03 00:59:46.419986 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id f73281be-aa8f-4388-b04b-d1ff4a79d24a picked up. Initiating work...
WARNING    | 2023-03-03 00:59:46.420967 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG      | 2023-03-03 00:59:46.421941 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:46.421941 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:47.304982 | worker.jobs.framework:submit_job:113 - Upload completed in 0.880648
INFO       | 2023-03-03 00:59:47.305518 | worker.jobs.framework:submit_job:137 - Submitted job with id f73281be-aa8f-4388-b04b-d1ff4a79d24a and contributed for 0.0
DEBUG      | 2023-03-03 00:59:48.886971 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.443463 (node: stablehorde:7001:4.1.0)
DEBUG      | 2023-03-03 00:59:49.456767 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 3.032s (Total Completed: 10002)
DEBUG      | 2023-03-03 00:59:49.477096 | worker.workers.framework:start_job:116 - New job processing
DEBUG      | 2023-03-03 00:59:49.477096 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG      | 2023-03-03 00:59:49.477096 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id 85b782b8-976f-41a2-9555-6632a031c83d picked up. Initiating work...
WARNING    | 2023-03-03 00:59:49.478076 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG      | 2023-03-03 00:59:49.479049 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:49.479049 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:50.471636 | worker.jobs.framework:submit_job:113 - Upload completed in 0.990169
INFO       | 2023-03-03 00:59:50.471636 | worker.jobs.framework:submit_job:137 - Submitted job with id 85b782b8-976f-41a2-9555-6632a031c83d and contributed for 0.0
DEBUG      | 2023-03-03 00:59:52.161064 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.660226 (node: stablehorde:7001:4.1.0)
DEBUG      | 2023-03-03 00:59:52.707828 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 3.235s (Total Completed: 10003)
DEBUG      | 2023-03-03 00:59:52.728832 | worker.workers.framework:start_job:116 - New job processing
DEBUG      | 2023-03-03 00:59:52.728832 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG      | 2023-03-03 00:59:52.728832 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id d6fc432c-b6d5-4d09-84f2-3869eed066cc picked up. Initiating work...
WARNING    | 2023-03-03 00:59:52.729812 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG      | 2023-03-03 00:59:52.730786 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-03-03 00:59:52.730786 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-03-03 00:59:54.332901 | worker.jobs.framework:submit_job:113 - Upload completed in 1.599558
INFO       | 2023-03-03 00:59:54.333429 | worker.jobs.framework:submit_job:137 - Submitted job with id d6fc432c-b6d5-4d09-84f2-3869eed066cc and contributed for 0.0
DEBUG      | 2023-03-03 00:59:55.536398 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.783143 (node: kai:7002:4.1.0)
WARNING    | 2023-03-03 00:59:55.536922 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 00:59:57.538836 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 4.813s (Total Completed: 10004)
DEBUG      | 2023-03-03 00:59:58.393343 | worker.jobs.poppers:horde_pop:39 - Job pop took 0.830176 (node: orc:7001:4.1.0)
WARNING    | 2023-03-03 00:59:58.393875 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:02.052249 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.630994 (node: kai:7002:4.1.0)
WARNING    | 2023-03-03 01:00:02.052249 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:05.051193 | worker.jobs.poppers:horde_pop:39 - Job pop took 0.973929 (node: troll:7001:4.1.0)
WARNING    | 2023-03-03 01:00:05.051721 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:08.425477 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.348741 (node: kai:7002:4.1.0)
WARNING    | 2023-03-03 01:00:08.425982 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:12.596920 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.144849 (node: thrr:7001:4.1.0)
WARNING    | 2023-03-03 01:00:12.597438 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:15.799604 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.176818 (node: kai:7002:4.1.0)
WARNING    | 2023-03-03 01:00:15.800137 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:18.975121 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.15014 (node: stablehorde:7001:4.1.0)
WARNING    | 2023-03-03 01:00:18.975121 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG      | 2023-03-03 01:00:22.196573 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.196226 (node: gremlin:7001:4.1.0)```
jug-dev commented 1 year ago

Maybe fixed by https://github.com/db0/AI-Horde/commit/6df4cb0d9549e2faf2ca704694c8dd6bda4efdc8