Closed jug-dev closed 1 year ago
This is pretty rare, but here's a log extract in any case:
DEBUG | 2023-02-07 00:26:55 | worker.workers.framework:start_job:120 - New job processing DEBUG | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: ProtoGen DEBUG | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:176 - img2img (ProtoGen) request with id 66080db1-fa83-41b0-a876-2f79abbb3c29 picked up. Initiating work... INFO | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:282 - Starting generation... DEBUG | 2023-02-07 00:26:57 | worker.jobs.poppers:horde_pop:36 - Job pop took 1.544195 INFO | 2023-02-07 00:26:57 | nataili.inference.compvis:generate:363 - [ProtoGen] (384x960) k_euler 100 steps (1/1) INFO | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:284 - Finished generation... DEBUG | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:324 - post-processing done... DEBUG | 2023-02-07 00:26:57 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool DEBUG | 2023-02-07 00:26:57 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 2.266s (Total Completed: 1121) DEBUG | 2023-02-07 00:26:57 | worker.workers.framework:start_job:120 - New job processing DEBUG | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix DEBUG | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c6e3d02a-a69a-4317-8457-6dc89818a9cb picked up. Initiating work... INFO | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:282 - Starting generation... DEBUG | 2023-02-07 00:26:58 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]> DEBUG | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:113 - Upload completed in 0.321775 WARNING | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 1/10) INFO | 2023-02-07 00:26:59 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1) DEBUG | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:113 - Upload completed in 0.183467 WARNING | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 2/10) DEBUG | 2023-02-07 00:27:01 | worker.jobs.poppers:horde_pop:36 - Job pop took 3.002054 DEBUG | 2023-02-07 00:27:02 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb INFO | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:284 - Finished generation... DEBUG | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:324 - post-processing done... DEBUG | 2023-02-07 00:27:02 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool DEBUG | 2023-02-07 00:27:02 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.766s (Total Completed: 1122) DEBUG | 2023-02-07 00:27:02 | worker.workers.framework:start_job:120 - New job processing DEBUG | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix DEBUG | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c7345143-5cbe-4648-9efe-b29222d20af1 picked up. Initiating work... INFO | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:282 - Starting generation... INFO | 2023-02-07 00:27:03 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1) DEBUG | 2023-02-07 00:27:03 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]> DEBUG | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:113 - Upload completed in 0.887986 WARNING | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 3/10) DEBUG | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:113 - Upload completed in 0.64281 INFO | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:137 - Submitted job with id c6e3d02a-a69a-4317-8457-6dc89818a9cb and contributed for 9.44 DEBUG | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:113 - Upload completed in 0.268584 WARNING | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 4/10) INFO | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:start_job:284 - Finished generation... DEBUG | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:start_job:324 - post-processing done... DEBUG | 2023-02-07 00:27:06 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool DEBUG | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]> DEBUG | 2023-02-07 00:27:06 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:113 - Upload completed in 0.887012 INFO | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:137 - Submitted job with id c7345143-5cbe-4648-9efe-b29222d20af1 and contributed for 9.44 DEBUG | 2023-02-07 00:27:07 | worker.jobs.poppers:horde_pop:36 - Job pop took 4.869675 DEBUG | 2023-02-07 00:27:07 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.922s (Total Completed: 1123) DEBUG | 2023-02-07 00:27:07 | worker.workers.framework:start_job:120 - New job processing DEBUG | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix DEBUG | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c24ecf25-2a1e-4184-880d-05a7872834ec picked up. Initiating work... INFO | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:282 - Starting generation... DEBUG | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb INFO | 2023-02-07 00:27:07 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1) DEBUG | 2023-02-07 00:27:08 | worker.jobs.framework:submit_job:113 - Upload completed in 0.660988 WARNING | 2023-02-07 00:27:08 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 5/10) DEBUG | 2023-02-07 00:27:10 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:113 - Upload completed in 0.551509 WARNING | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 6/10) INFO | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:start_job:284 - Finished generation... DEBUG | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:start_job:324 - post-processing done... DEBUG | 2023-02-07 00:27:11 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool DEBUG | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]> DEBUG | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:12 | worker.jobs.poppers:horde_pop:36 - Job pop took 4.343741 DEBUG | 2023-02-07 00:27:12 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.390s (Total Completed: 1124) DEBUG | 2023-02-07 00:27:12 | worker.workers.framework:start_job:120 - New job processing DEBUG | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix DEBUG | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id 0713a22c-67e6-425d-be40-30e075807337 picked up. Initiating work... INFO | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:282 - Starting generation... INFO | 2023-02-07 00:27:12 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1) DEBUG | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:113 - Upload completed in 0.287999 WARNING | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 7/10) DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:113 - Upload completed in 3.71467 INFO | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:137 - Submitted job with id c24ecf25-2a1e-4184-880d-05a7872834ec and contributed for 9.44 DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb INFO | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:start_job:284 - Finished generation... DEBUG | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:start_job:324 - post-processing done... DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool DEBUG | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]> DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:16 | worker.jobs.poppers:horde_pop:36 - Job pop took 3.883048 DEBUG | 2023-02-07 00:27:16 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 3.906s (Total Completed: 1125) DEBUG | 2023-02-07 00:27:16 | worker.workers.framework:start_job:120 - New job processing DEBUG | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Dreamlike Photoreal DEBUG | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Dreamlike Photoreal) request with id 04655444-5afc-471e-93f4-519c3943796d picked up. Initiating work... INFO | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:282 - Starting generation... DEBUG | 2023-02-07 00:27:16 | worker.jobs.framework:submit_job:113 - Upload completed in 0.85561 WARNING | 2023-02-07 00:27:16 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 8/10) DEBUG | 2023-02-07 00:27:17 | worker.jobs.framework:submit_job:113 - Upload completed in 1.835789 INFO | 2023-02-07 00:27:17 | worker.jobs.framework:submit_job:137 - Submitted job with id 0713a22c-67e6-425d-be40-30e075807337 and contributed for 9.44 INFO | 2023-02-07 00:27:17 | nataili.inference.compvis:generate:363 - [Dreamlike Photoreal] (512x768) k_dpmpp_sde 50 steps (1/1) DEBUG | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:113 - Upload completed in 0.222212 WARNING | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 9/10) DEBUG | 2023-02-07 00:27:19 | worker.jobs.poppers:horde_pop:36 - Job pop took 2.975233 DEBUG | 2023-02-07 00:27:20 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:21 | worker.jobs.framework:submit_job:113 - Upload completed in 0.611655 WARNING | 2023-02-07 00:27:21 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 10/10) DEBUG | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb DEBUG | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:113 - Upload completed in 0.20465 WARNING | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 11/10) ERROR | 2023-02-07 00:27:25 | worker.jobs.framework:submit_job:99 - Exceeded retry count 11 for job id 66080db1-fa83-41b0-a876-2f79abbb3c29. Aborting job! INFO | 2023-02-07 00:27:27 | worker.jobs.stable_diffusion:start_job:284 - Finished generation... DEBUG | 2023-02-07 00:27:27 | worker.jobs.stable_diffusion:start_job:324 - post-processing done... DEBUG | 2023-02-07 00:27:27 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool DEBUG | 2023-02-07 00:27:27 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 11.297s (Total Completed: 1126)
Closing as I have never seen this from the comfy worker.
This is pretty rare, but here's a log extract in any case: