Memory leak

[dm82m]

Hey @erikkastelec

I am not 100% sure, but it seems that we have a memory leak. Maybe also due to changes on wemportal.com

The last memory reduction was due to a server restart with disabled hass-WEM-Portal plugin. It SEEMS that the memory is not anymore increasing. Will keep an eye on it and give a new information in a couple of hours.

Best, Dirk

[erikkastelec]

Are you tracking the memory usage of the whole system or just Home Assistant? Will do a restart and see if I see something weird going on. Let me know if you find something.

[dm82m]

its the memory of the whole instance. BUT: with the plugin enabled memory usage increases and increases. with the plugin disabled the memory usage stays on the same level or stays at least in the same area...

[dm82m]

I would say its definitely a problem with the plugin. but as it worked in the past, I am pretty that its due to a change or maybe also due a "problem" on the wemportal.com side.

[erikkastelec]

Are you using only API, webscraping or both and are scan_interval and api_scan_interval set to default, if not, please let me know the values.

[dm82m]

  - platform: wemportal
    username: "my@mail.com"
    password: !secret wemportal

all default, nothing special.

will keep an eye on it and observe it a bit longer to ensure I am not blaming the plugin without a need.

[dm82m]

I found the issue. My heating system was disconnected somehow and this leaded the plug-in to increase and increase memory usage. I restarted my heating and guess the problem is gone so far. Maybe we should try to handle that situation. Depending on memory, it will sooner or later kill the pi.

[erikkastelec]

Will look into it. Great cache, I would have missed that as I have 32 GB ram and don't really notice small leaks.

[dm82m]

I was on a pi 3b+ with 1gb ram and moved to a pi4 with 8gb. will not run into a memory issue soon anymore, but other users with less ram and the plugin will run into that problem if the heating is somehow in an "not connected" state. this results in reaching the portal but not the heating. maybe you could simulate that by just removing the network cord to your heating and then observer what happens...

[dm82m]

Again the problem. Memory leak. Seems that the Webportal is unstable. But that should not lead to memory leak.

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: custom_components.wemportal
Source: custom_components/wemportal/wemportalapi.py:251 
Integration: wemportal 
First occurred: 11:01:02 (2 occurrences) 
Last logged: 11:33:38

Unexpected error fetching wem_portal_sensor data: 'Modules'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/wemportal/sensor.py", line 69, in async_update_data
    data = await hass.async_add_executor_job(api.fetch_data)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/wemportal/wemportalapi.py", line 69, in fetch_data
    self.fetch_api_data()
  File "/config/custom_components/wemportal/wemportalapi.py", line 100, in fetch_api_data
    self.getData()
  File "/config/custom_components/wemportal/wemportalapi.py", line 251, in getData
    for module in values["Modules"]:
KeyError: 'Modules'

and

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: custom_components.wemportal
Source: helpers/update_coordinator.py:193 
Integration: wemportal 
First occurred: 13:09:09 (1 occurrences) 
Last logged: 13:09:09

Timeout fetching wem_portal_sensor data

[dm82m]

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: custom_components.wemportal
Source: custom_components/wemportal/wemportalapi.py:194 
Integration: wemportal (documentation, issues) 
First occurred: 14:18:23 (1 occurrences) 
Last logged: 14:18:23

An error occurred while gathering parameters data. This issue should resolve by itself. If this problem persists, open an issue at https://github.com/erikkastelec/hass-WEM-Portal/issues'

[erikkastelec]

Will investigate and fix this as soon as I can. I am currently not at home so I can't efficiently test it. It looks to me that the problem is related to the API, which seems to be unreliable lately.

[mathiaslehmann]

Hello, I had similar problems recently. But I got it under control by adjusting my configuration.yaml. To be precise, the config for the logger component. I had no settings in the config before. I added the following block and since then I have no more memory problems. Maybe that will help somehow to find the problem.

[erikkastelec]

Interesting find. From this it looks like there is a probably a problem when logging warning messages. But I don't get it how logging alone could cause memory leak, so please let me know if the memory leak returns, while having logging set to error.

[mathiaslehmann]

Today I just happened to read about this problem here and thought I would share my findings. To narrow it down a bit more precisely, I made the following adjustment in the configuration of the logger. I am now observing how the RAM behaves.

[mathiaslehmann]

Hello, short status update. No memory problems so far. I adjusted the config again. I keep watching.

[mathiaslehmann]

Hello, quick update. So far no problems with the RAM. I adjusted the configuration as follows. The whole thing has been going on since yesterday afternoon. So far without any problems.

[mathiaslehmann]

Hello, another update. I don't have any memory problems.

[erikkastelec]

@dm82m is this still a problem?

[dm82m]

I close it as I am on raspberry with 8gb ram and even if it occurs it will take weeks till I run into a problem. And had no problems since I last commented this issue.