Open short-circuit opened 7 months ago
robballantyne
commented
7 months ago This may be because I didn't include a section to enable GPU. A fairly big oversight that I'll rectify soon.
Could you try appending the following to the compose file
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
short-circuit
commented
7 months ago Added the suggested configuration, did not seem to work sadly. Tried with already existing container and fresh install. Getting the same result, even with a fresh container (after wiping the volume).
robballantyne
commented
7 months ago Thanks for the update.
The ComfyUI process is ending too quickly. I need to change the supervisor startsecs to 5 (build/COPY_ROOT/etc/supervisor/supervisord/conf.d/comfyui.conf) to ensure its properly marked as a failure and maybe we will get some more useful information about why it's exiting. I haven't come across this bug previously.
I'll start a rebuild shortly - Sorry for the inconvenience
short-circuit
commented
7 months ago Tried building the image after changing the startsecs to 5, getting a build error:
Based on the logs, it was building fine. I am unable to find the point of failure.
robballantyne
commented
7 months ago What is your build environment? I have completed a successful local build prior to pushing changes on an intel i5, 4gb ram
short-circuit
commented
7 months ago Docker version 24.0.7, build afdd53b4e3 Docker Compose version 2.23.0
Hardware: Intel 13900K, 64GB RAM, nVidia 4090
OS: Archlinux Kernel 6.5.9 (build of 26th Oktober 2023)
Latest nVidia vulkan dev driver installed (through https://github.com/Frogging-Family/nvidia-all, works on other nVidia docker containers running pytorch)
Maybe I am overseeing something for building the image correctly?
robballantyne
commented
7 months ago I have no idea, unfortunately. To debug you can set +x on the build scripts to find exactly where the failure occurs.
I only build on Ubuntu (same as the GitHub runners) and there are no issues. I love Arch - Long time user, but it may have an unstable package causing trouble... Which one is a mystery but could also be causing the runtime problems.
Pushing a new build now.
maxfrei750
commented
7 months ago Hi @robballantyne, thanks for the awesome work! I think, I'm running into the same (or at least similar issue).
I already activated the GPU, as mentioned in https://github.com/ai-dock/comfyui/issues/8#issuecomment-1808931421.
I'm using an Nvidia GPU on Manjaro linux (basically arch).
robballantyne
commented
7 months ago I have re-worked some of the base images and an update is imminent. Hopefully it will address these issues as there is a new docker-compose.yaml and a slightly modified build process.
The new release includes some extra features (API requests to s3 storage which I need to be certain works properly before I commit) so it'll be early next week.
It may be worth investigating any difference in how Arch configures docker Vs Ubuntu. I use Mint and Ubuntu exclusively to stay close to my production environment and there are no build problems here.
I'd like it to work on Arch. I used it for many years after I gave up on Slackware so I'm very fond of it but it killed my productivity. If I do get some time I'll see what I can do to investigate but I can't give any ETA on that if the new release is still problematic.
maxfrei750
commented
6 months ago I just tested the current main branch and I think the problem still persists. Any ideas on how to investigate?
petra-pannia
commented
5 months ago I have a similar issue on such an AMD system, also with always encountering endless reconnects of ComfyUI,
Main system : ASRock DeskMini X300 AMD
APU : AMD R4750G Zen2 with 16GB UMA (graphical memories)
AMDGPU driver: 23.0.0-1 (X11)
ROCm : 5.7.1
Python : 3.11.6-1
Gross RAM's : 48GB DDR4
OS : Artix Linux amd64 with Linux kernel 6.1.70
Docker : 24.0.7-1 / Compose 2.24.3-1
UI browser : Brave 1.63.131-1However errors are still here and reconnections are still being occured, even already with 16GB UMA slightly more than required for properly functioning,
supervisor-1 | missing {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
supervisor-1 | left over keys: dict_keys(['betas', 'alphas_cumprod', 'alphas_cumprod_prev', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'log_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod', 'posterior_variance', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'model_ema.decay', 'model_ema.num_updates', 'cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
supervisor-1 | Requested to load SD1ClipModel
supervisor-1 | Loading 1 new model
supervisor-1 | 2024-01-30 00:33:07,515 INFO success: comfyui entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
supervisor-1 | 2024-01-30 00:33:08,473 INFO exited: comfyui (exit status 139; not expected)
supervisor-1 |
supervisor-1 | ==> /var/log/supervisor/supervisor.log <==
supervisor-1 | 2024-01-30 00:33:07,515 INFO success: comfyui entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
supervisor-1 | 2024-01-30 00:33:08,473 INFO exited: comfyui (exit status 139; not expected)
supervisor-1 | 2024-01-30 00:33:08,516 INFO spawned: 'comfyui' with pid 1392
supervisor-1 |
supervisor-1 | ==> /var/log/supervisor/caddy.log <==
supervisor-1 | {"level":"error","ts":1706574788.7762706,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"172.19.0.1","remote_port":"36190","client_ip":"172.19.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:8188","uri":"/ws?clientId=a99e72b16bc347b79664509660b01891","headers":{"Connection":["Upgrade"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Websocket-Key":["GU943067ry6S5AMfDZhRDw=="],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Origin":["http://localhost:8188"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"]}},"duration":0.000730051,"status":502,"err_id":"70zrr2js0","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
supervisor-1 | {"level":"error","ts":1706574789.0843854,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"172.19.0.1","remote_port":"36204","client_ip":"172.19.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:8188","uri":"/ws?clientId=a99e72b16bc347b79664509660b01891","headers":{"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.9"],"Origin":["http://localhost:8188"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Key":["qye4RtjldYYZoxfomOec7Q=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Pragma":["no-cache"],"Connection":["Upgrade"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"],"Upgrade":["websocket"]}},"duration":0.000468984,"status":502,"err_id":"p1e3thvc9","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
supervisor-1 | {"level":"error","ts":1706574789.3933089,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"172.19.0.1","remote_port":"36212","client_ip":"172.19.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:8188","uri":"/ws?clientId=a99e72b16bc347b79664509660b01891","headers":{"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Key":["P5VvvwE7iLde1fyvcBh0Gg=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Origin":["http://localhost:8188"],"Sec-Websocket-Version":["13"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade":["websocket"],"Connection":["Upgrade"]}},"duration":0.000509772,"status":502,"err_id":"psyuj9v7h","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}And finally, how to "download" for local SDiff models in a config file?
Using the shipped docker-compose file, the container does not seem to start correctly. Container logs spam the following lines over and over:
When trying to access
DOCKERMACHINE:8188caddy seems to work but the container then spits out the following error:A full log can be found underneath:
Full log
``` comfyui-supervisor-1 | You have no configured rclone remotes to be mounted comfyui-supervisor-1 | Looking for config.sh... comfyui-supervisor-1 | Not found comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/caddy.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/cloudflared.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/comfyui.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/logtail.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/logviewer.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/quicktunnel.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/rclone_mount.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/serverless.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/serviceportal.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/sshd.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Set uid to user 0 succeeded comfyui-supervisor-1 | 2023-11-13 13:55:00,642 INFO RPC interface 'supervisor' initialized comfyui-supervisor-1 | 2023-11-13 13:55:00,642 INFO supervisord started with pid 107 comfyui-supervisor-1 | 2023-11-13 13:55:01,644 INFO spawned: 'logtail' with pid 109 comfyui-supervisor-1 | 2023-11-13 13:55:01,645 INFO spawned: 'serverless' with pid 110 comfyui-supervisor-1 | 2023-11-13 13:55:01,645 INFO spawned: 'serviceportal' with pid 111 comfyui-supervisor-1 | 2023-11-13 13:55:01,646 INFO spawned: 'sshd' with pid 112 comfyui-supervisor-1 | 2023-11-13 13:55:01,646 INFO spawned: 'caddy' with pid 114 comfyui-supervisor-1 | 2023-11-13 13:55:01,647 INFO spawned: 'logviewer' with pid 115 comfyui-supervisor-1 | 2023-11-13 13:55:01,647 INFO spawned: 'comfyui' with pid 118 comfyui-supervisor-1 | Starting logtail service... comfyui-supervisor-1 | 2023-11-13 13:55:01,647 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:03,647 INFO success: serverless entered RUNNING state, process has stayed up for > than 2 seconds (startsecs) comfyui-supervisor-1 | Gathering logs...==> /var/log/config.log <== comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/sync.log <== comfyui-supervisor-1 | Mamba environments already present at /workspace/ comfyui-supervisor-1 | Linking mamba environments to /opt... comfyui-supervisor-1 | Creating symlink to /workspace/ComfyUI at /opt/ComfyUI comfyui-supervisor-1 | Creating symlink to /workspace/serverless at /opt/serverless comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/preflight.log <== comfyui-supervisor-1 | Looking for preflight.sh... comfyui-supervisor-1 | Updating ComfyUI (master)... comfyui-supervisor-1 | Already on 'master' comfyui-supervisor-1 | Your branch is up to date with 'origin/master'. comfyui-supervisor-1 | Already up to date. comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/debug.log <== comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/provisioning.log <== comfyui-supervisor-1 | Looking for provisioning.sh... comfyui-supervisor-1 | comfyui-supervisor-1 | ############################################## comfyui-supervisor-1 | # # comfyui-supervisor-1 | # Provisioning container # comfyui-supervisor-1 | # # comfyui-supervisor-1 | # This will take some time # comfyui-supervisor-1 | # # comfyui-supervisor-1 | # Your container will be ready on completion # comfyui-supervisor-1 | # # comfyui-supervisor-1 | ############################################## comfyui-supervisor-1 | comfyui-supervisor-1 | Updating node: https://github.com/ltdrdata/ComfyUI-Manager... comfyui-supervisor-1 | Already up to date. comfyui-supervisor-1 | Success comfyui-supervisor-1 | Downloading 3 model(s) to /opt/ComfyUI/models/checkpoints... comfyui-supervisor-1 | Downloading: https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/resolve/main/sd_xl_refiner_1.0.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading 4 model(s) to /opt/ComfyUI/models/controlnet... comfyui-supervisor-1 | Downloading: https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_canny-fp16.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_openpose-fp16.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/t2iadapter_canny-fp16.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/caddy.log <== comfyui-supervisor-1 | {"level":"info","ts":1699883703.7733493,"msg":"using provided configuration","config_file":"/opt/caddy/etc/Caddyfile","config_adapter":""} comfyui-supervisor-1 | {"level":"warn","ts":1699883703.774483,"logger":"http.auto_https","msg":"automatic HTTPS is completely disabled for server","server_name":"srv0"} comfyui-supervisor-1 | {"level":"warn","ts":1699883703.7744896,"logger":"http.auto_https","msg":"automatic HTTPS is completely disabled for server","server_name":"srv1"} comfyui-supervisor-1 | {"level":"warn","ts":1699883703.7744915,"logger":"http.auto_https","msg":"automatic HTTPS is completely disabled for server","server_name":"srv2"} comfyui-supervisor-1 | {"level":"warn","ts":1699883703.7744927,"logger":"http.auto_https","msg":"automatic HTTPS is completely disabled for server","server_name":"srv3"} comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Waiting for workspace sync... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/serverless.log <== comfyui-supervisor-1 | Refusing to start serverless worker without $SERVERLESS=true comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/serviceportal.log <== comfyui-supervisor-1 | Starting Service Portal... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/sshd.log <== comfyui-supervisor-1 | /root/.ssh/authorized_keys is not a public key file. comfyui-supervisor-1 | Skipping SSH server: No public key comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/caddy.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/cloudflared.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/comfyui.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/logtail.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/logviewer.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/quicktunnel.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/rclone_mount.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/serverless.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/serviceportal.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Included extra file "/etc/supervisor/supervisord/conf.d/sshd.conf" during parsing comfyui-supervisor-1 | 2023-11-13 13:55:00,641 INFO Set uid to user 0 succeeded comfyui-supervisor-1 | 2023-11-13 13:55:00,642 INFO RPC interface 'supervisor' initialized comfyui-supervisor-1 | 2023-11-13 13:55:00,642 INFO supervisord started with pid 107 comfyui-supervisor-1 | 2023-11-13 13:55:01,644 INFO spawned: 'logtail' with pid 109 comfyui-supervisor-1 | 2023-11-13 13:55:01,645 INFO spawned: 'serverless' with pid 110 comfyui-supervisor-1 | 2023-11-13 13:55:01,645 INFO spawned: 'serviceportal' with pid 111 comfyui-supervisor-1 | 2023-11-13 13:55:01,646 INFO spawned: 'sshd' with pid 112 comfyui-supervisor-1 | 2023-11-13 13:55:01,646 INFO spawned: 'caddy' with pid 114 comfyui-supervisor-1 | 2023-11-13 13:55:01,647 INFO spawned: 'logviewer' with pid 115 comfyui-supervisor-1 | 2023-11-13 13:55:01,647 INFO spawned: 'comfyui' with pid 118 comfyui-supervisor-1 | 2023-11-13 13:55:01,647 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:03,647 INFO success: serverless entered RUNNING state, process has stayed up for > than 2 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/provisioning.log <== comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/t2iadapter_openpose-fp16.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading 3 model(s) to /opt/ComfyUI/models/vae... comfyui-supervisor-1 | Downloading: https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/stabilityai/sdxl-vae/resolve/main/sdxl_vae.safetensors comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading 3 model(s) to /opt/ComfyUI/models/upscale_models... comfyui-supervisor-1 | Downloading: https://huggingface.co/ai-forever/Real-ESRGAN/resolve/main/RealESRGAN_x4.pth comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/FacehugmanIII/4x_foolhardy_Remacri/resolve/main/4x_foolhardy_Remacri.pth comfyui-supervisor-1 | comfyui-supervisor-1 | Downloading: https://huggingface.co/Akumetsu971/SD_Anime_Futuristic_Armor/resolve/main/4x_NMKD-Siax_200k.pth comfyui-supervisor-1 | comfyui-supervisor-1 | comfyui-supervisor-1 | Provisioning complete: Web UI will start now comfyui-supervisor-1 | comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: logtail entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: serviceportal entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: sshd entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: caddy entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: logviewer entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: logtail entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: serviceportal entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: sshd entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: caddy entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:06,649 INFO success: logviewer entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:55:11,647 INFO exited: serverless (exit status 0; expected) comfyui-supervisor-1 | 2023-11-13 13:55:11,649 INFO exited: sshd (exit status 0; expected) comfyui-supervisor-1 | 2023-11-13 13:55:11,647 INFO exited: serverless (exit status 0; expected) comfyui-supervisor-1 | 2023-11-13 13:55:11,649 INFO exited: sshd (exit status 0; expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/caddy.log <== comfyui-supervisor-1 | {"level":"error","ts":1699883724.371419,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"ACCESSMACHINE","remote_port":"48944","client_ip":"ACCESSMACHINE","proto":"HTTP/1.1","method":"GET","host":"DOCKERMACHINE:8188","uri":"/","headers":{"Authorization":[],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate"],"Dnt":["1"],"Connection":["keep-alive"]}},"duration":1.203366855,"status":502,"err_id":"admr5db0z","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"} comfyui-supervisor-1 | 2023-11-13 13:56:06,635 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 13:56:06,636 INFO spawned: 'comfyui' with pid 395 comfyui-supervisor-1 | 2023-11-13 13:56:06,650 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/serviceportal.log <== comfyui-supervisor-1 | INFO: Started server process [111] comfyui-supervisor-1 | INFO: Waiting for application startup. comfyui-supervisor-1 | INFO: Application startup complete. comfyui-supervisor-1 | INFO: Uvicorn running on http://127.0.0.1:11111 (Press CTRL+C to quit) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:56:06,635 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 13:56:06,636 INFO spawned: 'comfyui' with pid 395 comfyui-supervisor-1 | 2023-11-13 13:56:06,650 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:57:08,255 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:57:08,255 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 13:57:08,665 INFO spawned: 'comfyui' with pid 406 comfyui-supervisor-1 | 2023-11-13 13:57:08,685 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:57:08,665 INFO spawned: 'comfyui' with pid 406 comfyui-supervisor-1 | 2023-11-13 13:57:08,685 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:58:10,027 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:58:10,027 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 13:58:10,672 INFO spawned: 'comfyui' with pid 417 comfyui-supervisor-1 | 2023-11-13 13:58:10,690 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:58:10,672 INFO spawned: 'comfyui' with pid 417 comfyui-supervisor-1 | 2023-11-13 13:58:10,690 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 13:59:12,000 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:59:12,000 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 13:59:12,679 INFO spawned: 'comfyui' with pid 428 comfyui-supervisor-1 | 2023-11-13 13:59:12,700 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 13:59:12,679 INFO spawned: 'comfyui' with pid 428 comfyui-supervisor-1 | 2023-11-13 13:59:12,700 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 14:00:13,911 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 14:00:13,911 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 14:00:14,685 INFO spawned: 'comfyui' with pid 439 comfyui-supervisor-1 | 2023-11-13 14:00:14,707 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 14:00:14,685 INFO spawned: 'comfyui' with pid 439 comfyui-supervisor-1 | 2023-11-13 14:00:14,707 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 14:01:15,701 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 14:01:15,701 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 14:01:16,692 INFO spawned: 'comfyui' with pid 450 comfyui-supervisor-1 | 2023-11-13 14:01:16,718 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 14:01:16,692 INFO spawned: 'comfyui' with pid 450 comfyui-supervisor-1 | 2023-11-13 14:01:16,718 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | 2023-11-13 14:02:18,180 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | Success comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 14:02:18,180 INFO exited: comfyui (exit status 1; not expected) comfyui-supervisor-1 | 2023-11-13 14:02:18,698 INFO spawned: 'comfyui' with pid 461 comfyui-supervisor-1 | 2023-11-13 14:02:18,720 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/comfyui.log <== comfyui-supervisor-1 | Starting ComfyUI... comfyui-supervisor-1 | comfyui-supervisor-1 | ==> /var/log/supervisor/supervisor.log <== comfyui-supervisor-1 | 2023-11-13 14:02:18,698 INFO spawned: 'comfyui' with pid 461 comfyui-supervisor-1 | 2023-11-13 14:02:18,720 INFO success: comfyui entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) ```Any way to further diagnose the problem?