CoolMasterNet - Out of Memory exception after HA upgrade to 2024.9

sanketgroup commented 1 month ago

The problem

After upgrading home assistant to 2024.9, CoolMasterNet pro is going 'Out of memory'

See the attached photo of error shown on Coolmaster device screen.

It says (08054111->)ASRV:Out of memory <== 192.168.3.3:48358 (ASCL@08025F87)

This completely stops/hang the coolmaster device, and then I have to physically remove the power supply of Coolmaster device and restart it and Disable integration in HA.

Image 2024-09-09 at 17 54 10_39488988

What version of Home Assistant Core has the issue?

core-2024.9.1

What was the last working version of Home Assistant Core?

core-2024.8

What type of installation are you running?

Home Assistant OS

Integration causing the issue

CoolMasterNet

Link to integration documentation on our website

https://www.home-assistant.io/integrations/coolmaster/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 month ago

Hey there @onfreund, mind taking a look at this issue as it has been labeled with an integration (coolmaster) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `coolmaster` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign coolmaster` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

coolmaster documentation coolmaster source _{^{(message by IssueLinks)}}

ashalita commented 1 month ago

Hi @sanketgroup, can you provide more details?

Model and firmware
How often is this happening
Anything unique in the usage pattern (e.g. automations or special functions you use).
How is this device connected (not clear from the picture) - is this ethernet or serial connection?

sanketgroup commented 1 month ago

Model: CoolMaster Pro (Firmware, i not physically present at location, but can share you tmr)
How often: As soon as I Enable Integration, I get some feedback values to HA, and after 2-4 mins Coolmaster Pro goes Out of memory.
Nothing special, everything was working well for long untill 2024.9 update
Connected via Ethernet

amitfin commented 1 month ago

The firmware is also presented in the device's info page (for any device added by the integration). Can you please share a similar screenshot? (v1.2.0 is the latest according to this).
Please elaborate on "I get some feedback values to HA". Do you have anything to share (log entries, etc')?
Is 192.168.3.3 the address of HA or the CoolMaster device?

sanketgroup commented 1 month ago

following is screenshot of firmware, and below it is error in log. 192.168.3.3 is HA address.


Logger: homeassistant.components.coolmaster.coordinator
Source: helpers/update_coordinator.py:354
integration: CoolMasterNet (documentation, issues)
First occurred: 2:19:28 PM (94 occurrences)
Last logged: 3:52:30 PM

Unexpected error fetching coolmaster data
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/coolmaster/coordinator.py", line 30, in _async_update_data
    return await self._coolmaster.status()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pycoolmasternet_async/coolmasternet.py", line 97, in status
    for unit, key in await asyncio.gather(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pycoolmasternet_async/coolmasternet.py", line 122, in create
    swing_raw = ((await bridge._make_request(f"query {unit_id} s")).strip()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pycoolmasternet_async/coolmasternet.py", line 40, in _make_request
    response = await asyncio.wait_for(reader.readuntil(b"\n>"), self._read_timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/streams.py", line 657, in readuntil
    raise exceptions.IncompleteReadError(chunk, None)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of undefined expected bytes

sanketgroup commented 1 month ago

Any update? Pls resolve this as it has totally broke the integration after update

ashalita commented 4 weeks ago

@sanketgroup

how many units are in this deployment of CoolMaster?
can you acknowledge that there's nothing special in your setup (special CoolMaster firmware, intermediate interface between the Home Assistant device and the CoolMaster device, a Home Assistant automation that frequently accesses the CoolMaster integration, etc)?
as we don't have access to a CoolMasterPro device to test changes, would you be able to deploy a local custom package to assist analyzing the issue?

sanketgroup commented 4 weeks ago

There are around 48 units.

Yes there is nothing special in between HA and coolmaster. No script and automation on HA that controls coolmaster. Nothing changed on Coolmaster before this error, no firmware update or whatsoever.

Setup is very plain and simple, and it was working well since the beginning.

Yes, I can try to deploy custom package with your help.

Thanks

amitfin commented 3 weeks ago

@sanketgroup , from where the firmware was taken, and how was it installed? The latest official version is 1.2, while the device has 1.3.

sanketgroup commented 3 weeks ago

It was not taken from anywhere, it was already pre installed when I received the device.

On Sun, 15 Sept 2024 at 12:41 am, Amit @.***> wrote:

@sanketgroup , from where the firmware was taken, and how was it installed? The latest official version is 1.2, while the device has 1.3.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

sanketgroup commented 2 weeks ago

@ashalita can we have some solution on this please? because integration is not working at all.

ashalita commented 2 weeks ago

@sanketgroup can you verify that the issue does not occur in 2024.8? Also, can you access the coolmaster device through a terminal, e.g, with "nc 10102" (in mac or linux) or "telnet 10102" (in windows)?

sanketgroup commented 2 weeks ago

I downgraded same HA to 2024.8 and everything works well with same CoolMasterNet integration. Kept under observation for 48 hrs and nothing went wrong.

So, it confirms that the issue started from 2024.9.

Yes I can do telnet, and also have their CoolAutomation utility installed on windows.

ashalita commented 2 weeks ago

Hi @sanketgroup, thanks for performing these checks.

We believe a change to pycoolmasternet_async (the library powering the integration) may have caused this issue. Specifically, a change made to how the coolmaster prompt is parsed may cause the integration to send more requests than usual, overloading your unit. To test this hypothesis we’ve created a branch that disables this behavior, it would be great if you can install it as a custom component and report back if the issue is gone. To do so, you can follow these steps:

1) Copy the following files locally: https://github.com/ashalita/core/tree/prompt-wait-dep/homeassistant/components/coolmaster. An easy way to do so is to paste the link above here https://download-directory.github.io and grab the generated archive 2) ssh into the homeassistant device, under "config/" create "custom_components/coolmaster/" and upload the files from step 1 into that location 3) Restart homeassistant (Settings -> System -> Restart button (top right))-> “Restart Home Assistant”) 4) Verify the custom integration is indeed in use (Settings -> "Device and Service" -> "CoolMasterNet" should say "version 7.7.7", “custom integration”) 5) Please report back if this version performs better on your device.

sanketgroup commented 1 week ago

I followed your instruction and getting following error in start up.

Logger: homeassistant.setup
Source: setup.py:334
First occurred: 1:21:48 PM (1 occurrences)
Last logged: 1:21:48 PM

Setup failed for custom integration 'coolmaster': Unable to import component: No module named 'custom_components.coolmaster.pycoolmasternet_async_prompt'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
    self._component_future.result()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
    comp = await self.hass.async_add_import_executor_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/coolmaster/__init__.py", line 10, in <module>
    from .pycoolmasternet_async_prompt import CoolMasterNet
ModuleNotFoundError: No module named 'custom_components.coolmaster.pycoolmasternet_async_prompt'

ashalita commented 1 week ago

@sanketgroup can you verify that the pycoolmasternet_async_prompt directory was copied under custom_components/coolmaster/ and that it has read permissions?

sanketgroup commented 1 week ago

my bad, the directory was missing and I fixed and restarted HA.

coming to main issue: New custom component is installed and there is no error on Device anymore. The device has been up for the last 7 hours and has not thrown any error on the device so far.

ashalita commented 1 week ago

Thanks for the update @sanketgroup. We'll work on a solution to the prompt parsing issue, in the meantime you can keep the custom integration or delete it and revert back to 2024.8.

home-assistant / core