Integration causes a constant and abnormal increasing of CPU and RAM Memory usage.

Pippo1989 commented 1 year ago

Description

I've successfully integrated four Tapo C420 cameras in Home Assistant. The cameras don't have an assigned IP, since they use the Tapo Hub to communicate with the app. So, the integration was made by using the Hub IP address, and nothing went wrong. Despite that, I discovered that the integration causes a constant and enormous increasing in CPU and RAM memoru usage; as a result, the overall performance of my HA decreased, with many response delays. Even turning on a light with a zigbee button became a struggle. The CPU usage never increased over the 5% threshold, and now it's over 20% and still increasing hour by hour. The only thing that sets the things right is rebooting the system, but then the CPU start increasing again. Same for the RAM Memory. I tried reinstalling the integration and disabling all the non-necessary entities, but the problem seems to be still there (I'm just using the motion sensor and the battery entities, no streaming). My setup Is pretty clean and light, i have very few integrations and a low number of devices.

Reproduction Steps

It was described above.

Expected behavior

I shouldn't experience an increase of CPU/Memory and heavy lags.

If applicable, add error logs.

///

Device Firmware

1.2.4 Build 20221208 rel.73688

Using stream component

No

Does camera work via official integrations?

N/A

Camera has all attributes filled out in developer tools

Yes

HASS Environment

I'm using HASSIO on a Raspberry PI4 with 4GB RAM Memory.

Search for similar issues

Yes

Additional information

No response

JurajNyiri commented 1 year ago

Looks like a memory / cpu cycle leak. I am not experiencing this with multiple c200s so I cannot test this. I don't really see a reason why this would be happening since the battery cameras actually do less, maybe some initialization / update logic is wrong since there is the difference.

Pippo1989 commented 1 year ago

I have also noticed that the integration requires a lot of time to initialize (more than 90 seconds - in my case, this is the integration that requires more time: in second place, there is Alexa Media Player, about 12 seconds) and, in my logs, i get these warnings:

Setup of select platform tapo_control is taking over 10 seconds. 08:56:53 – (AVVISO) Seleziona Setup of number platform tapo_control is taking over 10 seconds. 08:56:53 – (AVVISO) Numero Setup of light platform tapo_control is taking over 10 seconds. 08:56:53 – (AVVISO) Luce Setup of button platform tapo_control is taking over 10 seconds. 08:56:53 – (AVVISO) Pulsante Setup of switch platform tapo_control is taking over 10 seconds. 08:56:53 – (AVVISO) Interruttore

Anyway, thanks in advance for any help or suggestions... I'd like not to have to return my Tapo cameras for not being able to solve this isssue :(

Pippo1989 commented 1 year ago

I dont know if I can be of any help, but I'm getting 2 kind of errors in my logs, concerning this integrations, with a lot occurrences. Could that explain the memory leak? Is there any way to fix them?

Here you can find the error messages:

Logger: custom_components.tapo_control Source: custom_components/tapo_control/init.py:289 Integration: Tapo: Cameras Control (documentation, issues) First occurred: 20:53:28 (248 occurrences) Last logged: 22:56:55

Unexpected error fetching Tapo resource status data: <pytapo.Tapo object at 0x7f8796fa30> Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 225, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 181, in _async_update_data return await self.update_method() File "/config/custom_components/tapo_control/init.py", line 289, in async_update_data ] = updateDataForAllControllers[tapoController] KeyError: <pytapo.Tapo object at 0x7f8796fa30>

Logger: custom_components.tapo_control Source: custom_components/tapo_control/init.py:285 Integration: Tapo: Cameras Control (documentation, issues) First occurred: 20:58:20 (227 occurrences) Last logged: 22:51:53

HTTPSConnectionPool(host='192.168.1.100', port=443): Max retries exceeded with url: /stok=c08e99acee619920a804e8df8f33f559/ds (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7a38d540>: Failed to establish a new connection: [Errno 111] Connection refused'))
HTTPSConnectionPool(host='192.168.1.100', port=443): Max retries exceeded with url: /stok=b0b51d4df9bb28d90af096e99daf7db9/ds (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f795b3b20>: Failed to establish a new connection: [Errno 111] Connection refused'))
HTTPSConnectionPool(host='192.168.1.100', port=443): Max retries exceeded with url: /stok=d799b35bde8b319fafe26e6b540db875/ds (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f71b3dc90>: Failed to establish a new connection: [Errno 111] Connection refused'))
HTTPSConnectionPool(host='192.168.1.100', port=443): Max retries exceeded with url: /stok=277559316db9db54c4425d06c2f858c4/ds (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f71b3cd00>: Failed to establish a new connection: [Errno 111] Connection refused'))
HTTPSConnectionPool(host='192.168.1.100', port=443): Max retries exceeded with url: /stok=c08e99acee619920a804e8df8f33f559/ds (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f71b3d1e0>: Failed to establish a new connection: [Errno 111] Connection refused'))

JurajNyiri commented 1 year ago

The error shows your camera hub is refusing the connection. You probably need to fix the network.

Pippo1989 commented 1 year ago

I have the requested TCP/lan ports already open, and the integration, however, works without problem? What could It be?

Anyway, I'm reporting this error as well:

Logger: homeassistant.helpers.entity Source: helpers/entity.py:543 First occurred: 01:58:03 (1 occurrences) Last logged: 01:58:03

Updating state for sensor.ingresso_battery (<class 'custom_components.tapo_control.sensor.TapoBatterySensor'>) took 0.546 seconds. Please report it to the custom integration author.

JurajNyiri commented 1 year ago

The update is fine.

The issue here is that the camera is refusing the connection sometimes, which might be the reason of the performance issues (which should be fixed).

As for the help with the network that is something you will need to figure out and fix.

Pippo1989 commented 1 year ago

Hi,

Indeed, I have noticed that the Motion Detection entity becomes unavailable from time to time, for a short amount of time (I have enabled only that entity and the battery entity).

Could you give me any hint about how to investigate? I really don't know where to start. Thanks in advance!

Filippo

Il sab 14 gen 2023, 12:36 Juraj Nyíri @.***> ha scritto:

The update is fine.

The issue here is that the camera is refusing the connection sometimes, which might be the reason of the performance issues (which should be fixed).

As for the help with the network that is something you will need to figure out and fix.

— Reply to this email directly, view it on GitHub https://github.com/JurajNyiri/HomeAssistant-Tapo-Control/issues/278#issuecomment-1382719122, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXQLB672GMQP2B3RCIUG2GLWSKFU7ANCNFSM6AAAAAATZVMUUY . You are receiving this because you authored the thread.Message ID: @.***>

Pippo1989 commented 1 year ago

Hi @JurajNyiri,

I'm literally going crazy because of this issue.

I'm concluding that the problem is that your integration works fine with any Tapo camera, except for those that needs an hub.

I alredy have a Tapo C110 perfecty working with Home Assistant: all the troubles arose with Tapo C420 and its hub.

I tried:

checking the Port-Forwarding / Virtual Server configuration on my router;
monitoring the quality of the wifi signal (which is "strong" for every camera);
updating my router firmware;
resetting the hub;
removing and reinstalling the integration (many times);
removing the Ipv4 and Ipv6 firewall protections,

but nothing changed.

I guess I'll have to return all my battery Tapo camera and move to another brand, unless you can provide for some other way to try to fix the issue.

JurajNyiri commented 1 year ago

Hello, please enable debug logs and send them. Ideally also add comments to where did you see the performance decrease.

Pippo1989 commented 1 year ago

Hi,

I've started the debug logs for the integration, restarted Home assistant and stopped the debug about 30 minutes later. All I got from the logs is here (as already posted):

2023-01-15 12:58:32.648 WARNING (MainThread) [homeassistant.components.switch] Setup of switch platform tapo_control is taking over 10 seconds. 2023-01-15 12:58:32.654 WARNING (MainThread) [homeassistant.components.button] Setup of button platform tapo_control is taking over 10 seconds. 2023-01-15 12:58:32.658 WARNING (MainThread) [homeassistant.components.light] Setup of light platform tapo_control is taking over 10 seconds. 2023-01-15 12:58:32.659 WARNING (MainThread) [homeassistant.components.number] Setup of number platform tapo_control is taking over 10 seconds. 2023-01-15 12:58:32.662 WARNING (MainThread) [homeassistant.components.select] Setup of select platform tapo_control is taking over 10 seconds.

The performance decrease starts with just enabling the integration, it is enough to make CPU and Memory begin their climb to the top.

It's likely that, later or sooner, I'll get the connections issue reports as the logs previously posted.

JurajNyiri commented 1 year ago

That means you have not enabled debug logs properly.

logger:
  default: critical
  logs:
    custom_components.tapo_control: debug
    homeassistant.components.onvif: debug

Pippo1989 commented 1 year ago

Critical logs TAPO.txt.txt

Sorry, I did not know I could edit the logger. I've attached the critical logs (about 10 minutes from HA startup). If you need more, I have more than 30 minutes of logs, but the file was too big.

P.S. "Ingresso", "Giardino", "Auto" and "Cancello" are the names of each camera.

JurajNyiri commented 1 year ago

Thank you for submitting the logs. I will take a look a it after my vacation during February.

Pippo1989 commented 1 year ago

Well, thank you. I have 15 days left to return my cameras; so, unless you will able to take a quick look to provide for a possible solution soon, I'll have to move to another brand anyway.

Have a nice day.

Filippo

Il dom 15 gen 2023, 17:29 Juraj Nyíri @.***> ha scritto:

Thank you for submitting the logs. I will take a look a it after my vacation during February.

— Reply to this email directly, view it on GitHub https://github.com/JurajNyiri/HomeAssistant-Tapo-Control/issues/278#issuecomment-1383193589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXQLB6Z6TL2XODO3M7AZD5LWSQQYDANCNFSM6AAAAAATZVMUUY . You are receiving this because you authored the thread.Message ID: @.***>

djtail commented 1 year ago

having the same issues, when adding c100 and c200 to my installation, the cpu goes up to 100% and then makes the pi freeze.

no problems with c310

Adivor commented 1 year ago

having the same issues, when adding c100 and c200 to my installation, the cpu goes up to 100% and then makes the pi freeze.

no problems with c310

same with c200, freeze and raspberry restart required

JurajNyiri commented 1 year ago

I checked the logs and nothing out of the ordinary in them. I think as a next step someone experiencing the issue will need to debug this in the code.

Pippo1989 commented 1 year ago

CPU

Hi, there must be something to fix for sure. In the pic you can see the growth of CPU when the integration is enabled. I disabled it when it passed 50%. It's so frustrating :(

Pippo1989 commented 1 year ago

1.log 2.txt

Hi, I've uploaded about 17 minutes (part 1 & part 2) of logs obtained by using the Profiler integration, hoping that could help to find what the problem is caused by. First I've enabled the Profiler, then Tapo Cameras control and eventually restarted Home Assistant without any further operations to let the Profiler record.

By viewing the recordings, I noticed many redundant updates, constantly repeated in a short time, for the cameras, Is that normal? Could that be the reason why the CPU usage grows relentlessly?

Jedediah04 commented 1 year ago

Hello, can confirm I have the same issue with C420S. CPU load shows spikes that are increasing over time and RAM usage keeps increasing.

Disabling the integration immediately shows a near 0% CPU load.

Running home assistant on docker and I will be happy to provide logs.

djtail commented 1 year ago

when i switched to 32bits installation of HA the problem stopped for me

JurajNyiri commented 1 year ago

Thank you for all the logs, it helped me figure out the issue.

Could you please test with 5.0.0.beta.6?

You will need to enable beta releases in HACS or install manually.

Pippo1989 commented 1 year ago

Hi,

thanks for your commitment, it seems to work now! The memory leak seems gone, at least during the first 15-20 minutes of usage.

I'm submitting about 10 minutes of logs, recorded after HA was restarted, if useful.

home-assistant_profiler_2023-03-21T12-51-21.316Z.log

JurajNyiri / HomeAssistant-Tapo-Control