search

FL550 / dwd_weather

Deutscher Wetterdienst integration for Home-Assistant
MIT License
188 stars 12 forks source link

High ram usage #129

Closed MauiR closed 3 months ago

MauiR commented 3 months ago

Version of home_assistant

Core: 2024.6.2 Supervisor: 2024.06.0 Operating System: 12.3 Frontend: 20240610.0

Version of the custom_component

Version 2.1.3

Describe the bug

Integration consumes a lot of RAM. Up to one gigabyte of RAM is required when the data is downloaded. The hourly query feature is activated. The RAM is only released again after a delay Here is a picture of the ram consumption in correlation with the installation of the addon.

image

Debug log

FL550 commented 3 months ago

Thanks for reporting. This is probably related to issue #122.

I know technically where this issue originates. However I am not sure if I can solve this. I'll have to look a bit deeper into this if I find the time.

The issue with the hourly update is, that (as stated in the info text of the setting) this requires to load a large file compared to the normal 6 hour update. This file is a compressed file of roughly 900MB. This file have to be extracted and scanned for the relevant part, which is only a fraction of the total data. There is some streaming mechanism implemented to reduce the memory load, however it seems it is not working in all cases.

MauiR commented 3 months ago

Thanks for reporting. This is probably related to issue #122.

I know technically where this issue originates. However I am not sure if I can solve this. I'll have to look a bit deeper into this if I find the time.

The issue with the hourly update is, that (as stated in the info text of the setting) this requires to load a large file compared to the normal 6 hour update. This file is a compressed file of roughly 900MB. This file have to be extracted and scanned for the relevant part, which is only a fraction of the total data. There is some streaming mechanism implemented to reduce the memory load, however it seems it is not working in all cases.

Thank you for the clarification. Does this mean that if you deactivate the hourly query, the ram is usage is reduced? If the problem is known and has been dealt with in another issue, this issue can be closed.

FL550 commented 3 months ago

Does this mean that if you deactivate the hourly query, the ram is usage is reduced?

Yes, the RAM usage should then be only a few MB.

frostbyte-ninja commented 3 months ago

@FL550 It seems there's another aspect to this issue. Can you confirm that all data is thoroughly cleaned up after processing? I've experienced several system freezes on a Raspberry Pi running Home Assistant with the hourly update feature. While most of the RAM was eventually freed, about 100MB persisted after each update, causing the system to crash every 2-3 days due to full RAM.