When running a ComfyUI workflow on cog v0.9.13 and upwards, the workflow would seem to randomly just stop and there would be no output. On adding error logging a broken pipe error was reported.
The error would always stop at the point where it would attempt to print or log something (like tqdm or otherwise).
ComfyUI is running in a subprocess (and the server it runs has some async methods), and the model gets updates from webhooks, which it prints. But at the same time Comfy is printing it's own outputs (like inference steps) which are also useful.
When running a ComfyUI workflow on cog v0.9.13 and upwards, the workflow would seem to randomly just stop and there would be no output. On adding error logging a broken pipe error was reported.
The error would always stop at the point where it would attempt to print or log something (like tqdm or otherwise).
ComfyUI is running in a subprocess (and the server it runs has some async methods), and the model gets updates from webhooks, which it prints. But at the same time Comfy is printing it's own outputs (like inference steps) which are also useful.
It seems to be related to this change in cog: https://github.com/replicate/cog/pull/1802
The fix here is to capture all stdout/stderr from the subprocess and log it separately.