search

crystian / ComfyUI-Crystools

A powerful set of tools for ComfyUI
MIT License
546 stars 26 forks source link

Hi there. Not working for me, still the same error even after all other plugins disabled, fresh ComfyUI install, etc. #63

Closed Ainaemaet closed 2 weeks ago

Ainaemaet commented 1 month ago

Hi there. Not working for me, still the same error even after all other plugins disabled, fresh ComfyUI install, etc.

I am referencing as a new issue as the 'it works for me' approach and reinstalling hasn't helped; and even after my post yesterday, the issue remains closed (not sure why).

Could it be something in the fact that Crystools hasn't been updated in months, but ComfyUI has experienced many updates in that time? I'm sorry I can't be of more help, but I can give you whatever information you need about my setup!

As I mentioned previously, a ComfyUI update a couple weeks ago broke many 3rd party nodes - but apparently the comfyanonymous changed some things again in order to afford better compatibility, but still many that don't work.

I've seen mention of a few other people having trouble with crystools in particular, ~but can't find them anymore~ (found one in the response to my comment here! 🙃).

File "/home/myML/anaconda3/envs/comfyui/lib/python3.12/site-packages/pynvml/nvml.py", line 1744, in wrapper return res.decode() ^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte

Does the utf-8 error at the end of the traceback I posted mean the files aren't utf-8 encoded? Sorry, I'm not very well versed in python outside of setting up and running programs and some very minor scripting.

I'm running on WSL2 Ubuntu20.04 if that helps - though I never had any problems with Crystools in particular until a ComfyUI update a week or so ago!

(ty for the help).

Originally posted by @Ainaemaet in https://github.com/crystian/ComfyUI-Crystools/issues/61#issuecomment-2146661635

Artur3d commented 1 month ago

Same here, fresh conda, fresh Comfyui, can't load the node, error : nvml.py line 1744

Ainaemaet commented 1 month ago

Same here, fresh conda, fresh Comfyui, can't load the node, error : nvml.py line 1744

That's exactly what I'm seeing; does the rest of your traceback look like this?

Traceback (most recent call last):
File "/home/myML/ComfyUI/nodes.py", line 1879, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/init.py", line 24, in
from .server import *
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/server/init.py", line 1, in
from .monitor import *
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/server/monitor.py", line 4, in
from ..general import cmonitor
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/general/init.py", line 1, in
from .monitor import *
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/general/monitor.py", line 66, in
cmonitor = CMonitor(1, True, True, True, True, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/general/monitor.py", line 20, in init
self.hardwareInfo = CHardwareInfo(switchCPU, switchGPU, switchHDD, switchRAM, switchVRAM)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/general/hardware.py", line 44, in init
self.GPUInfo = CGPUInfo()
^^^^^^^^^^
File "/home/myML/ComfyUI/custom_nodes/ComfyUI-Crystools/general/gpu.py", line 45, in init
gpuName = pynvml.nvmlDeviceGetName(deviceHandle)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myML/anaconda3/envs/comfyui/lib/python3.12/site-packages/pynvml/nvml.py", line 1744, in wrapper
return res.decode()
^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
Artur3d commented 1 month ago

@Ainaemaet , yes same :

[Crystools INFO] CPU: AMD Ryzen 9 3900X 12-Core Processor - Arch: x86_64 - OS: Linux 5.15.146.1-microsoft-standard-WSL2
[Crystools INFO] GPU/s:
Traceback (most recent call last):
  File "/home/arox/ComfyUI/nodes.py", line 1879, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/__init__.py", line 24, in <module>
    from .server import *
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/server/__init__.py", line 1, in <module>
    from .monitor import *
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/server/monitor.py", line 4, in <module>
    from ..general import cmonitor
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/general/__init__.py", line 1, in <module>
    from .monitor import *
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/general/monitor.py", line 66, in <module>
    cmonitor = CMonitor(1, True, True, True, True, True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/general/monitor.py", line 20, in __init__
    self.hardwareInfo = CHardwareInfo(switchCPU, switchGPU, switchHDD, switchRAM, switchVRAM)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/general/hardware.py", line 44, in __init__
    self.GPUInfo = CGPUInfo()
                   ^^^^^^^^^^
  File "/home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools/general/gpu.py", line 45, in __init__
    gpuName = pynvml.nvmlDeviceGetName(deviceHandle)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arox/miniconda3/envs/comfyui/lib/python3.12/site-packages/pynvml/nvml.py", line 1744, in wrapper
    return res.decode()
           ^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte

Cannot import /home/arox/ComfyUI/custom_nodes/ComfyUI-Crystools module for custom nodes: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
ComfyUI-Manager (V2.36)
ComfyUI Revision: 2222 [809cc85a] | Released on '2024-06-02'
NVidia 3090 RTX / 555.85
Python 3.11
Win11 / WSL2

I'm not a dev but I did some research in particular GPT, maybe the problem does not come from the Crystian Node, but maybe a bad UTF-8 configuration of the computer :

(ChatGPT4o)
To troubleshoot this issue, you can try a few things:

• Check Data Source: Ensure that the data being read by pynvml is indeed encoded in UTF-8. It's possible that the data source has a different encoding.

• Handle Encoding: You might need to handle the encoding explicitly if it's not UTF-8. Python provides various methods for handling different encodings, such as decode() and encode().

• Update Libraries: Ensure that all relevant libraries (pynvml, etc.) are up to date. Sometimes, encoding-related bugs are fixed in newer versions of libraries.

• Check System Locale: If your system's locale is not set properly, it could lead to encoding issues. Ensure that your system's locale is set to a UTF-8 compatible setting.

• Inspect Data: If possible, inspect the data causing the issue to understand its encoding and content better.

Still investigating, but some help would be really appreciated .

Ainaemaet commented 1 month ago

@Artur3d Ok with the help of ChatGPT I managed to get it fixed. I've submitted a pull request to @crystian

You can find the fix in my fork https://github.com/Ainaemaet/ComfyUI-Crystools in /general/gpu.py

Artur3d commented 1 month ago

@Ainaemaet I try your fixe but have errors :

[Crystools INFO] Crystools version: 1.12.0
[Crystools INFO] CPU: AMD Ryzen 9 3900X 12-Core Processor - Arch: x86_64 - OS: Linux 5.15.146.1-microsoft-standard-WSL2
[Crystools INFO] GPU/s:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
[Crystools INFO] 0) Unknown GPU (decoding error)
[Crystools INFO] NVIDIA Driver: 555.85
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.03              Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:0B:00.0  On |                  N/A |
|  0%   51C    P8             36W /  350W |     854MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        29      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+
Ainaemaet commented 1 month ago

@Artur3d

Hi there, could you please try first: sudo apt update sudo apt upgrade?

and then re-install directory from my fork (git clone https://github.com/Ainaemaet/ComfyUI-Crystools in custom_nodes folder, + pip install -r requirements.txt)?

If that still doesn't work, be sure your GPU drivers are all updated, and you can also try: uninstall/reinstall pynvml, or upgrading: pip install --upgrade pynvml

You can try setting environment variable (please note if you just type it this way into your terminal, it will only last until the next deactivation of your env - but if after testing it works, you can add to /envs/comfyui/etc/conda/activate.d/env_vars.sh or to .bashrc or .zshrc to make it permanent: 'export PYTHONIOENCODING=utf-8'

Double check that your ComfyUI-Crystools/general/gpu.py looks like this: 

import torch
import pynvml
import comfy.model_management
from ..core import logger

class CGPUInfo:
    """
    This class is responsible for getting information from GPU (ONLY).
    """
    cuda = False
    pynvmlLoaded = False
    cudaAvailable = False
    torchDevice = 'cpu'
    cudaDevice = 'cpu'
    cudaDevicesFound = 0
    switchGPU = True
    switchVRAM = True
    switchTemperature = True
    gpus = []
    gpusUtilization = []
    gpusVRAM = []
    gpusTemperature = []

    def __init__(self):
        try:
            pynvml.nvmlInit()
            self.pynvmlLoaded = True
        except Exception as e:
            self.pynvmlLoaded = False
            logger.error('Could not init pynvml.' + str(e))

        if self.pynvmlLoaded and pynvml.nvmlDeviceGetCount() > 0:
            self.cudaDevicesFound = pynvml.nvmlDeviceGetCount()

            logger.info(f"GPU/s:")

            # for simulate multiple GPUs (for testing) interchange these comments:
            # for deviceIndex in range(3):
            #   deviceHandle = pynvml.nvmlDeviceGetHandleByIndex(0)
            for deviceIndex in range(self.cudaDevicesFound):
                deviceHandle = pynvml.nvmlDeviceGetHandleByIndex(deviceIndex)

                try:
                    gpuName = pynvml.nvmlDeviceGetName(deviceHandle).decode('utf-8', errors='ignore')
                except UnicodeDecodeError as e:
                    gpuName = 'Unknown GPU (decoding error)'
                    print(f"UnicodeDecodeError: {e}")

                logger.info(f"{deviceIndex}) {gpuName}")

                self.gpus.append({
                    'index': deviceIndex,
                    'name': gpuName,
                })

                # same index as gpus, with default values
                self.gpusUtilization.append(True)
                self.gpusVRAM.append(True)
                self.gpusTemperature.append(True)

            self.cuda = True
            logger.info(f'NVIDIA Driver: {pynvml.nvmlSystemGetDriverVersion()}')
        else:
            logger.warn('No GPU with CUDA detected.')

        try:
            self.torchDevice = comfy.model_management.get_torch_device_name(comfy.model_management.get_torch_device())
        except Exception as e:
            logger.error('Could not pick default device.' + str(e))

        self.cudaDevice = 'cpu' if self.torchDevice == 'cpu' else 'cuda'
        self.cudaAvailable = torch.cuda.is_available()

        if self.cuda and self.cudaAvailable and self.torchDevice == 'cpu':
            logger.warn('CUDA is available, but torch is using CPU.')

    def getInfo(self):
        logger.debug('Getting GPUs info...')
        return self.gpus

    def getStatus(self):
        # logger.debug('CGPUInfo getStatus')
        gpuUtilization = -1
        gpuTemperature = -1
        vramUsed = -1
        vramTotal = -1
        vramPercent = -1

        gpuType = ''
        gpus = []

        if self.cudaDevice == 'cpu':
            gpuType = 'cpu'
            gpus.append({
                'gpu_utilization': 0,
                'gpu_temperature': 0,
                'vram_total': 0,
                'vram_used': 0,
                'vram_used_percent': 0,
            })
        else:
            gpuType = self.cudaDevice

            if self.pynvmlLoaded and self.cuda and self.cudaAvailable:
                # for simulate multiple GPUs (for testing) interchange these comments:
                # for deviceIndex in range(3):
                #   deviceHandle = pynvml.nvmlDeviceGetHandleByIndex(0)
                for deviceIndex in range(self.cudaDevicesFound):
                    deviceHandle = pynvml.nvmlDeviceGetHandleByIndex(deviceIndex)

                    gpuUtilization = 0
                    vramPercent = 0
                    vramUsed = 0
                    vramTotal = 0
                    gpuTemperature = 0

                    # GPU Utilization
                    if self.switchGPU and self.gpusUtilization[deviceIndex]:
                        try:
                            utilization = pynvml.nvmlDeviceGetUtilizationRates(deviceHandle)
                            gpuUtilization = utilization.gpu
                        except Exception as e:
                            if str(e) == "Unknown Error":
                                logger.error('For some reason, pynvml is not working in a laptop with only battery, try to connect and turn on the monitor')
                            else:
                                logger.error('Could not get GPU utilization.' + str(e))

                            logger.error('Monitor of GPU is turning off (not on UI!)')
                            self.switchGPU = False

                    # VRAM
                    if self.switchVRAM and self.gpusVRAM[deviceIndex]:
                        # Torch or pynvml?, pynvml is more accurate with the system, torch is more accurate with comfyUI
                        memory = pynvml.nvmlDeviceGetMemoryInfo(deviceHandle)
                        vramUsed = memory.used
                        vramTotal = memory.total

                        # device = torch.device(gpuType)
                        # vramUsed = torch.cuda.memory_allocated(device)
                        # vramTotal = torch.cuda.get_device_properties(device).total_memory

                        vramPercent = vramUsed / vramTotal * 100

                    # Temperature
                    if self.switchTemperature and self.gpusTemperature[deviceIndex]:
                        try:
                            gpuTemperature = pynvml.nvmlDeviceGetTemperature(deviceHandle, 0)
                        except Exception as e:
                            logger.error('Could not get GPU temperature. Turning off this feature. ' + str(e))
                            self.switchTemperature = False

                    gpus.append({
                        'gpu_utilization': gpuUtilization,
                        'gpu_temperature': gpuTemperature,
                        'vram_total': vramTotal,
                        'vram_used': vramUsed,
                        'vram_used_percent': vramPercent,
                    })

        return {
            'device_type': gpuType,
            'gpus': gpus,
        }

with the 

try:
      gpuName = pynvml.nvmlDeviceGetName(deviceHandle).decode('utf-8', errors='ignore')
except UnicodeDecodeError as e:
      gpuName = 'Unknown GPU (decoding error)'
      print(f"UnicodeDecodeError: {e}")

replacing the previous: 'gpuName = pynvml.nvmlDeviceGetName(deviceHandle)' that was at line 45 in gpu. py

Other than that, I don't know what else you can do?

Artur3d commented 1 month ago

Hi @Ainaemaet thank you very much for your time ! It start working with no more steps than that :

sudo apt update
sudo apt upgrade?

and then re-install directory from my fork (git clone https://github.com/Ainaemaet/ComfyUI-Crystools in custom_nodes folder, + pip install -r requirements.txt)?

If that still doesn't work, be sure your GPU drivers are all updated, and you can also try:
uninstall/reinstall pynvml, or upgrading:
pip install --upgrade pynvml

Still have errors on loading , but everything work fine ! ♥

[Crystools INFO] Crystools version: 1.12.0
[Crystools INFO] CPU: AMD Ryzen 9 3900X 12-Core Processor - Arch: x86_64 - OS: Linux 5.15.146.1-microsoft-standard-WSL2
[Crystools INFO] GPU/s:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
[Crystools INFO] 0) Unknown GPU (decoding error)
[Crystools INFO] NVIDIA Driver: 555.85
Ainaemaet commented 1 month ago

@Artur3d

I still get those errors too, but everything works just fine now. :)

"UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte [Crystools INFO] 0) Unknown GPU (decoding error)"

I might tackle the reason why later and try to come up with a solution - just because I really like the way things look with no errors, even if they are completely functional lol.

Good_stuff, I'm glad I could be of help.

PurplefinNeptuna commented 1 month ago

gpuopenanalytics/pynvml#53 it's NVML fault, they said it fixed in next driver release

crystian commented 2 weeks ago

try again please! And thanks @Ainaemaet for the PR