Comfy process gets killed

[signorinileandro]

Hi! I'm having this issue where the process gets killed in the face restore node. Is this the issue you were talking about in the pull request?

Logs:

`Length of mel chunks: 2419 Load model from: /home/leandro/ComfyUI/custom_nodes/ComfyUI_wav2lip/Wav2Lip/checkpoints/wav2lip_gan.pth Load checkpoint from: /home/leandro/ComfyUI/custom_nodes/ComfyUI_wav2lip/Wav2Lip/checkpoints/wav2lip_gan.pth 0%| | 0/19 [00:00<?, ?it/s]Downloading: "https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth" to /home/leandro/.cache/torch/hub/checkpoints/s3fd-619a316812.pth 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 85.7M/85.7M [00:10<00:00, 8.38MB/s] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:25<00:00, 1.86s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 19/19 [01:16<00:00, 4.01s/it] out_images len = 2419 Starting restore_face with codeformer_fidelity: 0.5 Downloading: "https://github.com/xinntao/facexlib/releases/download/v0.1.0/detection_Resnet50_Final.pth" to /home/leandro/ComfyUI/models/facedetection/detection_Resnet50_Final.pth

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 104M/104M [00:03<00:00, 35.9MB/s] Downloading: "https://github.com/sczhou/CodeFormer/releases/download/v0.1.0/parsing_parsenet.pth" to /home/leandro/ComfyUI/models/facedetection/parsing_parsenet.pth

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 81.4M/81.4M [00:02<00:00, 31.6MB/s] Killed `

[ShmuelRonen]

Try this node with Image without wav2lip. are the same error?

[signorinileandro]

No, without the Wav2Lip node, there's no error. Could it be because Wav2Lip outputs images, and the face restore node only accepts a single image?

[ShmuelRonen]

Try the wav2lip without the face restoration node to see if it make's output

[signorinileandro]

Yes it does

[signorinileandro]

[signorinileandro]

The same happens when you connect the video combine node to wav2lip, comfyui crashes

[ShmuelRonen]

The error message "Killed" typically indicates that the process was terminated by the operating system, most likely due to running out of memory (RAM). This can happen when the process requires more memory than is available on the system. Here are a few steps to address and troubleshoot this issue:

1. Check System Resources:

   • Monitor your system's memory usage while running the process. Use tools like htop or free -m on Linux to see how much RAM is being used and if it's hitting the limit.

2. Reduce Memory Usage:

   • Optimize the process to use less memory. This can include reducing the batch size or using a more efficient model if possible.
   • If you are working with large images or videos, try processing them in smaller chunks or reducing their resolution.

3. Increase Swap Space:

   • Increasing the swap space on your system can help mitigate out-of-memory errors. While this won't be as fast as physical RAM, it can provide a buffer to prevent the process from being killed.
   • On Linux, you can add swap space with the following commands:

     sudo fallocate -l 4G /swapfile
     sudo chmod 600 /swapfile
     sudo mkswap /swapfile
     sudo swapon /swapfile

   • Make sure to add the swap space to /etc/fstab to make it persistent across reboots.

4. Check for Memory Leaks:

   • Ensure there are no memory leaks in the code that could cause it to consume more memory over time. Use tools like valgrind or memory profilers to identify and fix leaks.

5. Upgrade Hardware:

   • If feasible, consider adding more physical RAM to your system. This is a more permanent solution if the process consistently requires more memory than is currently available.

6. Log Details:

   • Add more logging to understand which part of the process is consuming the most memory. This can help identify specific areas to optimize.

7. Divide the Task:

   • If possible, divide the task into smaller, more manageable parts that can be processed sequentially instead of all at once.

Here is a summary of your log with the critical parts highlighted:

Length of mel chunks: 2419
Load model from: /home/leandro/ComfyUI/custom_nodes/ComfyUI_wav2lip/Wav2Lip/checkpoints/wav2lip_gan.pth
Load checkpoint from: /home/leandro/ComfyUI/custom_nodes/ComfyUI_wav2lip/Wav2Lip/checkpoints/wav2lip_gan.pth
0%| | 0/19 [00:00<?, ?it/s]Downloading: "https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth" to /home/leandro/.cache/torch/hub/checkpoints/s3fd-619a316812.pth
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 85.7M/85.7M [00:10<00:00, 8.38MB/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:25<00:00, 1.86s/it]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 19/19 [01:16<00:00, 4.01s/it]
out_images len = 2419
Starting restore_face with codeformer_fidelity: 0.5
Downloading: "https://github.com/xinntao/facexlib/releases/download/v0.1.0/detection_Resnet50_Final.pth" to /home/leandro/ComfyUI/models/facedetection/detection_Resnet50_Final.pth

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 104M/104M [00:03<00:00, 35.9MB/s]
Downloading: "https://github.com/sczhou/CodeFormer/releases/download/v0.1.0/parsing_parsenet.pth" to /home/leandro/ComfyUI/models/facedetection/parsing_parsenet.pth

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 81.4M/81.4M [00:02<00:00, 31.6MB/s]
Killed

In this log, the process is killed right after the downloading and loading of large models, which suggests that the memory usage peaks at this point. Implementing some of the suggested steps should help mitigate this issue.

[signorinileandro]

I don't think it's a memory issue, as I tested it again on a computer with a RTX 4090 and 32gb of ram. With which workflow can I produce a lip sync video?