search

Haidra-Org / AI-Horde-Worker

This repo turns your PC into a AI Horde worker node
GNU Affero General Public License v3.0
245 stars 70 forks source link

Job already submitted error #39

Closed jug-dev closed 1 year ago

jug-dev commented 1 year ago

This is pretty rare, but here's a log extract in any case:

DEBUG      | 2023-02-07 00:26:55 | worker.workers.framework:start_job:120 - New job processing
DEBUG      | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: ProtoGen
DEBUG      | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:176 - img2img (ProtoGen) request with id 66080db1-fa83-41b0-a876-2f79abbb3c29 picked up. Initiating work...
INFO       | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG      | 2023-02-07 00:26:57 | worker.jobs.poppers:horde_pop:36 - Job pop took 1.544195
INFO       | 2023-02-07 00:26:57 | nataili.inference.compvis:generate:363 - [ProtoGen] (384x960) k_euler 100 steps (1/1)
INFO       | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG      | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG      | 2023-02-07 00:26:57 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-02-07 00:26:57 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 2.266s (Total Completed: 1121)
DEBUG      | 2023-02-07 00:26:57 | worker.workers.framework:start_job:120 - New job processing
DEBUG      | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG      | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c6e3d02a-a69a-4317-8457-6dc89818a9cb picked up. Initiating work...
INFO       | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG      | 2023-02-07 00:26:58 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG      | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:113 - Upload completed in 0.321775
WARNING    | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 1/10)
INFO       | 2023-02-07 00:26:59 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG      | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:113 - Upload completed in 0.183467
WARNING    | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 2/10)
DEBUG      | 2023-02-07 00:27:01 | worker.jobs.poppers:horde_pop:36 - Job pop took 3.002054
DEBUG      | 2023-02-07 00:27:02 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
INFO       | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG      | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG      | 2023-02-07 00:27:02 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-02-07 00:27:02 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.766s (Total Completed: 1122)
DEBUG      | 2023-02-07 00:27:02 | worker.workers.framework:start_job:120 - New job processing
DEBUG      | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG      | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c7345143-5cbe-4648-9efe-b29222d20af1 picked up. Initiating work...
INFO       | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
INFO       | 2023-02-07 00:27:03 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG      | 2023-02-07 00:27:03 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG      | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:113 - Upload completed in 0.887986
WARNING    | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 3/10)
DEBUG      | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:113 - Upload completed in 0.64281
INFO       | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:137 - Submitted job with id c6e3d02a-a69a-4317-8457-6dc89818a9cb and contributed for 9.44
DEBUG      | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:113 - Upload completed in 0.268584
WARNING    | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 4/10)
INFO       | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG      | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG      | 2023-02-07 00:27:06 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG      | 2023-02-07 00:27:06 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:113 - Upload completed in 0.887012
INFO       | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:137 - Submitted job with id c7345143-5cbe-4648-9efe-b29222d20af1 and contributed for 9.44
DEBUG      | 2023-02-07 00:27:07 | worker.jobs.poppers:horde_pop:36 - Job pop took 4.869675
DEBUG      | 2023-02-07 00:27:07 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.922s (Total Completed: 1123)
DEBUG      | 2023-02-07 00:27:07 | worker.workers.framework:start_job:120 - New job processing
DEBUG      | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG      | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c24ecf25-2a1e-4184-880d-05a7872834ec picked up. Initiating work...
INFO       | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG      | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
INFO       | 2023-02-07 00:27:07 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG      | 2023-02-07 00:27:08 | worker.jobs.framework:submit_job:113 - Upload completed in 0.660988
WARNING    | 2023-02-07 00:27:08 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 5/10)
DEBUG      | 2023-02-07 00:27:10 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:113 - Upload completed in 0.551509
WARNING    | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 6/10)
INFO       | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG      | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG      | 2023-02-07 00:27:11 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG      | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:12 | worker.jobs.poppers:horde_pop:36 - Job pop took 4.343741
DEBUG      | 2023-02-07 00:27:12 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.390s (Total Completed: 1124)
DEBUG      | 2023-02-07 00:27:12 | worker.workers.framework:start_job:120 - New job processing
DEBUG      | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG      | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id 0713a22c-67e6-425d-be40-30e075807337 picked up. Initiating work...
INFO       | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
INFO       | 2023-02-07 00:27:12 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG      | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:113 - Upload completed in 0.287999
WARNING    | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 7/10)
DEBUG      | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:113 - Upload completed in 3.71467
INFO       | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:137 - Submitted job with id c24ecf25-2a1e-4184-880d-05a7872834ec and contributed for 9.44
DEBUG      | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
INFO       | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG      | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG      | 2023-02-07 00:27:15 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG      | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:16 | worker.jobs.poppers:horde_pop:36 - Job pop took 3.883048
DEBUG      | 2023-02-07 00:27:16 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 3.906s (Total Completed: 1125)
DEBUG      | 2023-02-07 00:27:16 | worker.workers.framework:start_job:120 - New job processing
DEBUG      | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Dreamlike Photoreal
DEBUG      | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Dreamlike Photoreal) request with id 04655444-5afc-471e-93f4-519c3943796d picked up. Initiating work...
INFO       | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG      | 2023-02-07 00:27:16 | worker.jobs.framework:submit_job:113 - Upload completed in 0.85561
WARNING    | 2023-02-07 00:27:16 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 8/10)
DEBUG      | 2023-02-07 00:27:17 | worker.jobs.framework:submit_job:113 - Upload completed in 1.835789
INFO       | 2023-02-07 00:27:17 | worker.jobs.framework:submit_job:137 - Submitted job with id 0713a22c-67e6-425d-be40-30e075807337 and contributed for 9.44
INFO       | 2023-02-07 00:27:17 | nataili.inference.compvis:generate:363 - [Dreamlike Photoreal] (512x768) k_dpmpp_sde 50 steps (1/1)
DEBUG      | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:113 - Upload completed in 0.222212
WARNING    | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 9/10)
DEBUG      | 2023-02-07 00:27:19 | worker.jobs.poppers:horde_pop:36 - Job pop took 2.975233
DEBUG      | 2023-02-07 00:27:20 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:21 | worker.jobs.framework:submit_job:113 - Upload completed in 0.611655
WARNING    | 2023-02-07 00:27:21 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 10/10)
DEBUG      | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG      | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:113 - Upload completed in 0.20465
WARNING    | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds...  (Retry 11/10)
ERROR      | 2023-02-07 00:27:25 | worker.jobs.framework:submit_job:99 - Exceeded retry count 11 for job id 66080db1-fa83-41b0-a876-2f79abbb3c29. Aborting job!
INFO       | 2023-02-07 00:27:27 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG      | 2023-02-07 00:27:27 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG      | 2023-02-07 00:27:27 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG      | 2023-02-07 00:27:27 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 11.297s (Total Completed: 1126)
jug-dev commented 1 year ago

Closing as I have never seen this from the comfy worker.