2023.9.1 update KO: very very slow, impossible to navigate

jerome6994 commented 10 months ago

The problem

Good evening

Since update 2023.9.0 and even with 2023.9.1 I have been experiencing slowness in using all of Home Assistant and not just on the dashboard!

So I created 2 VMs on proxmox (6vCPU and 6GB of Ram) On a VM I put HAOS in version 10.5 On the second VM I put HAOS in version 10.4

I compared the two without any particular addition of integration or other! The elements are identical on both SAMBA installations, MariaDB and phpMyAdmin

Latency is visible on version 10.5 and it is more obvious when you restart.

Then I connected my Mosquitto to the two VMs (233 devices and 1900 entities). I observe the same latency between the two VMs! I keep the same gap in reactivity!

When I run the configuration check on the VM with HAOS 10.5 I get the green result after 7 seconds on the HAOS 10.4 VM it's instantaneous!

In short, I know it's not computer research but I'm not a computer scientist, just tests of uses, recipes, etc.

But for me, version 10.5 greatly degrades my environment which is very demanding in terms of devices, integration and database of around 1.8 GB

Thanking you for your help on the subject

What version of Home Assistant Core has the issue?

core-2023.9.1

What was the last working version of Home Assistant Core?

core-2023.8.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

My HOAS production (That I can't update to 2023.9.1 otherwise very slow more than 30 seconds to change pages) :

System Information

version	core-2023.8.4
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.4
os_name	Linux
os_version	6.1.45
arch	x86_64
timezone	Europe/Paris
config_dir	/config

Home Assistant Community Store

Home Assistant Cloud

Home Assistant Supervisor

Dashboards

Recorder

My HAOS test OS10.5 (Which shows very strong slowness) :

System Information

version	core-2023.9.1
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.5
os_name	Linux
os_version	6.1.45
arch	x86_64
timezone	Europe/Paris
config_dir	/config

Home Assistant Cloud

Home Assistant Supervisor

Dashboards

Recorder

My HOAS Test OS 10.4 (which seems to work)

System Information

version	core-2023.9.1
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.5
os_name	Linux
os_version	6.1.39
arch	x86_64
timezone	Europe/Paris
config_dir	/config

Home Assistant Cloud

Home Assistant Supervisor

Dashboards

Recorder

joostlek commented 10 months ago

Have you noticed an increase in memory usage or cpu usage?

deanfourie1 commented 10 months ago

Same issue for me! Frontend is almost unusable and can take up to 1 minute to become usable after opening the HA app. This has even crashed my Proxmox host on multiple occasions.

Updated to 2023.9.1 but no changes.

Also, everytime I open the app now, I receive the error

Failed to call service system_log/write. unknown error

joostlek commented 10 months ago

@jerome6994 do you also see this in the logs?

jerome6994 commented 10 months ago

@joostlek

Good morning It is true that I have not looked at this particular point of overloading either the Ram or the CPU. did you have in mind the load indicated in HA or that of the proxmox VM? Because there is an important difference between the two, that I am sure, many of us have noticed.

without doing anything on my production HAOS version 2023.8.4: in HA I have: CPU: between 4% and 7%, for Ram: 2GB out of 6GB On the proxmox VM: CPU: between 5% and 6%, for the Ram: 5GB out of 6GB (this is 1GB more than at the beginning of September but kernel updates on the proxmox side, can this have an effect!)

After update 2023.9.1: no change in the load on HA or in Proxmox.

In parallel on the two test VMs mounted yesterday I have the following elements.

On my HAOS 10.4 test VM (MQTT 236 devices integration): in HA I have: CPU: between 0% and 1%, for Ram: 0.9GB out of 6GB On the proxmox VM: CPU: 1.5%, for Ram: 1.9GB out of 6GB

On my HAOS 10.5 test VM (MQTT 236 devices integration): in HA I have: CPU: between 0% and 1%, for Ram: 2GB out of 6GB On the proxmox VM: CPU: 1.7%, for Ram: 1.9GB out of 6GB

I don't see the error: Failed to call service system_log/write. unknown error in the logs though.

I have several errors and warnings but I don't know how to correct them! The information presented does not speak to me enough yet.

joostlek commented 10 months ago

At this point I am curious what causes this. The other person also mentioned that they use proxmox for their HA, so I'm currently maybe thinking about a relation between these two. If it would behave like this on all proxmox instances this wouldn't be the only issue on GitHub since 2023.9.

Since it's proxmox, are you maybe able to create a new instance and check if that has the same behaviour out of the box. If that's the case -> there could be a relation between proxmox and HA. Then slowly add more integrations until you're at the same point. Try to monitor CPU load and memory usage in the process and I would expect at some point it might change.

If it doesn't change it has to do with your old instance (if we're at that point we'll have to look what's going wrong).

joostlek commented 10 months ago

@deanfourie1 can you create a separate issue? Maybe this issue will also help you, but this error may be a different problem that also is occurring.

bramkragten commented 10 months ago

Can you share the logs you get? Can you also check the browser console for errors? Thanks

deanfourie1 commented 10 months ago

@deanfourie1 can you create a separate issue? Maybe this issue will also help you, but this error may be a different problem that also is occurring.

Sure, but the slowness issue still relates to me, as the frontend is so slow its almost unusable.

joostlek commented 10 months ago

We might have an idea, please create a new issue with both backend logs, and frontend logs

jerome6994 commented 10 months ago

At this point I am curious what causes this. The other person also mentioned that they use proxmox for their HA, so I'm currently maybe thinking about a relation between these two. If it would behave like this on all proxmox instances this wouldn't be the only issue on GitHub since 2023.9.

Since it's proxmox, are you maybe able to create a new instance and check if that has the same behaviour out of the box. If that's the case -> there could be a relation between proxmox and HA. Then slowly add more integrations until you're at the same point. Try to monitor CPU load and memory usage in the process and I would expect at some point it might change.

If it doesn't change it has to do with your old instance (if we're at that point we'll have to look what's going wrong).

Thank you already for your feedback and your involvement, but also to all those who improve and advance HA.

I have already created 2 VMs on the new 2023.9.1 instance On these two instances I have:

Samba
VSCode
MariaDB
phpMyAdmin
MQTT

One like

Another one like :

configuration check takes 7 seconds longer than with HAOS version 10.4

on version HAOS 10.5 the configuration check takes 7 seconds longer than with version 10.4 of HAOS

I don't know if the question of logs concerns me but yes I can give them if necessary but since I really don't always have my nose in it, is there any personal or secret information that could be visible?

joostlek commented 10 months ago

Out of the box It shouldn't log any sensitive info. It doesn't hurt to check.

jerome6994 commented 10 months ago

Here is the log of my current HA without the update already. home-assistant_2023-09-10T14-02-09.122Z.log

I restart a VM for a while to get the log with the update

jerome6994 commented 10 months ago

Good morning

I come to bring news of my work and tests from the weekend.

I have an enphase integration which is not the core one (D7 firmware for quite some time already). this integration was in error and trying to reset all the time. So I decided last night to turn it off. this solved a very large part of the slowness but there are still some remaining.

Perhaps this comes from the connection method set up by the integration but the problem was noted in line with a message from them on my mobile application. This morning a message gave the problem as resolved and I restarted the integration, direct connection and latency still not present!

My personal subject is, in my opinion, corrected. there remain more general slownesses

EDIT : On the other hand, what I can't explain is the difference in speed between version 10.4 and 10.5 of HAOS on my two test VMs freshly prepared for the occasion!

majorhector commented 9 months ago

Same problem here like @deanfourie1 . I have updated to 2023.9.2. After that the frontend give me the message "Failed to call service system_log/write. unknown error" and the system is unstable and unusable. This happens everytime I open the app. After 30 second the message disappear, everithing works fine.

I have HassOS on a Raspberry Pi 4B Supervisor 2023.9.2 OS 10.5 Frontend 20230911.0

cpuks commented 9 months ago

Same problems on VM in proxmox - I've doubled the resources but it's still laggy - 8 GB RAM from 4 and 4 cores from 2.

joostlek commented 9 months ago

Okay so what I am seeing is a lot of issues and all of you use Proxmox. If all instances with proxmox were failing this issue would've probably exploded.

Last week I was suspecting one thing, since 2023.9 the backend receives frontend errors to log. But that uses the system_log/write service. So if that service couldn't be called it would throw another error that it couldn't log the error, but to log that error it threw another error since. ... you get the idea. Bram patched this in https://github.com/home-assistant/frontend/pull/17893 (2023.9.2).

So I am now curious if these issues are coming from the fact that HA itself is slow. Does it use way more resources? Since some of you are also mentioning this frontend issue I just told about, I am getting curious if it's the application itself being slow, or that the frontend is slow, since for example @majorhector says that the application is unusable but it everything still works fine.

I don't have a proxmox instance, never worked with it so I don't know the specifics. But together we can maybe pinpoint what's wrong.

cpuks commented 9 months ago

A post above mine is about RPi 4B so bare metal no proxmox involved... As for my setup I didn't change anything except applying updates - nothing new in meantime - it's really poor performance considering 4 CPU cores of i5-1240p and 8 GB RAM

joostlek commented 9 months ago

Oh lol, my head skipped that part.

@majorhector Could you open a new issue for your problem?

@cpuks Can you share more about usages? Did you see any increase in CPU usage, memory usage? I want to find out where the problem is.

majorhector commented 9 months ago

Oh lol, my head skipped that part.

@majorhector Could you open a new issue for your problem?

@cpuks Can you share more about usages? Did you see any increase in CPU usage, memory usage? I want to find out where the problem is.

Yes. I'll open a new issue. Anyway I think that with Bram's patch I could solve the problem.

joostlek commented 9 months ago

Yes but you're already running his patch so there must be something else. It was included in 2023.9.2.

majorhector commented 9 months ago

Right! I misread, sorry 😅 I have confused the versions

jerome6994 commented 9 months ago

Good evening Come on, I'll come back and give information on my problem, our concerns.

A little detail of my configuration to help too. I have a large installation it seems to me: 2468 active entities and 2885 non-active. Some examples : I have typical integrations MQTT (238 devices, 1895 entities) Unifi Network (562 devices, 892 entities), Promox VE (20 services, 491 entities), Enphase (16 devices, 31 entities)

My Proxmox 8 VM is 6 core, 6GB of Ram and I do not notice any changes in load depending on the different versions.

I carried out filtering under zigbee2mqtt (not in HA in a separate docker) to limit the load on my broker (not in HA in a separate docker). I deactivated, deleted what I could delete in HA, purged my database which is 2GB (many sockets connected and I log the energy and power) Besides, if in front of each entity we had a check box to activate/deactivate and history Yes/No it would simplify mass processing ;) Today I cleared Chrome's cache and that seems pretty good too.

This Friday, I thought Chrome had a new performance feature: I went to look and this option was disabled! Once active I really found the best in the evening.

On my iPhone, I don't see these slowdowns! However, I find them on Firefoxe!

So isn't it related to the browser having particular options? Or maps that consume too much browser resources? but how to measure this?

I really find that the most fluid and quickest version is: Home Assistant 2023.9.2 Supervisor 2023.09.2 Operating System 10.4 User interface: 20230911.0 - latest

there slowdown: Home Assistant 2023.9.2 Supervisor 2023.09.2 Operating System 10.5 User interface: 20230911.0 - latest

but I don't know what and where, really sorry

This is the work of the week

joostlek commented 9 months ago

And to clarify, what do you mean with slowdown? Is it mostly controlling entities that goes slow? (So from the frontend to the controlling device) or are we talking about that moving from 1 page to the other in frontend goes slow?

The first would indicate something in the backend, the latter would mean in the frontend and could differ per browser.

jerome6994 commented 9 months ago

Good morning

So I don't really know how to answer this question.

The slowdown appeared on the frontend, in the navigation. But when I activated a Hue lamp for example (or another) via a Lexman button it was possible that I would see the status of the lamp a few seconds later or even not at all! I had to restart Mosquitto or zigbee2mqtt! So for me it's a bit of both but it really showed up in the PC browser and not on mobile

cpuks commented 9 months ago

As others mentioned hard to explain but the frontend experience is really laggy - even doubling resources didn't help - if I want to display entities, or check something then I came to a point when it's faster to close the tab or HA companion app, and reopen and go straight to place I want as otherwise, it takes more time to wait for UI to respond rather than reopening UI again. Don't know when that slowdown exactly happened as I updated HASS OS to 10.5 and HA 2023.9.0 at more or less the same time, most the things for me are done under the hood but when I have to do something through UI or check something it's really a downgrade especially considering I didn't change anything in meantime and my setup is overkill anyway - HA running on two P cores of i5-1240p with 4 GB RAM - databases are written to outside docker containers.

Things I've checked - RAM consumption and CPU utilization are on the same level as before - the culprit for me is frontend.

joostlek commented 9 months ago

if @jerome6994 can also find his problem in this explanation we can keep it in one issue.

So let's assume this to be a frontend issue. Have you tried disabling custom cards? Can you check the logs of your browser? Did you upgrade to 2023.9.2 (it has a fix for a log loop where it tried to log that it can't log, and loops and that could probably mess up your browser).

deanfourie1 commented 9 months ago

if @jerome6994 can also find his problem in this explanation we can keep it in one issue.

So let's assume this to be a frontend issue. Have you tried disabling custom cards? Can you check the logs of your browser? Did you upgrade to 2023.9.2 (it has a fix for a log loop where it tried to log that it can't log, and loops and that could probably mess up your browser).

Just FYI, 2023.9.2 did not fix this log error issue for me.

joostlek commented 9 months ago

Yes, just wanted to double check with him as well that he was experiencing the same issue.

Same for you, can you disable custom cards and check logs in the browser?

cpuks commented 9 months ago

Just had a power outage a while ago and now things are coming back and digging through frontend now it's really a pain - performance is like on RPi 3 - something is really odd - I turn off the entity and literally see the movement of a checkbox, then the icon is changing colours from on to off like I have PC from 1999 again.

It took me almost a minute to toggle one relay and I had a second tab opened on system resources - max out was 18% of CPU usage even though it took ages to do this via UI.

joostlek commented 9 months ago

I am definetly leaning towards a frontend issue, do you see anything in the frontend logs?

cpuks commented 9 months ago

Nothing special to be honest - just regular timeouts during the startup process etc - one HACS addon stopped working when I went to 2023.9 but I removed it as it was spamming logs. I've been running 2023.9.2 for a good week now and problems still persist - as I said it's faster to close a tab and reopen it than wait for data to load - if needed I can record how things are working to prove it's really slow on frontend side while resources usage is really low not to be said none. home-assistant_2023-09-21T11-20-53.666Z.log Logs attached

My setup is fairly complex but all resoucre heavy things are outside in docker containers running in the same proxmox system to HA is just sending data via an internal virtualized 100Gbe switch.

mhoogenbosch commented 9 months ago

I have been running into same issues. A few days ago I (no real reason related to this issue) deleted the theme I was using (iOS themes) and my setup feels a lot faster. Haven’t seen the write log errors anymore.

could just be coincidence but wanted to share non the less.

sam-ward commented 9 months ago

I was having the same issue with the front end slowing to a crawl around the 2023.6, 2023.7 release. Couldn't work out why as it was affecing the system menu's as well as the dashboards. and even with a blank dashboard it was terrible.

Turns out I had forgotten that I had turned on the BLE device tracker, which had been running for 6 months or so. My known_devices.yaml had grown to 9MB and had over 80000 lines in it. Now i'm still not sure what the relationship was between the known_devices and the front end (given there didnt appear to be any extra websocket traffic), but removing the file (and disbaling the integration) brought the frontend back to full speed.

So just check you you don't have a rouge integration running.

sam-ward commented 9 months ago

Sorry hit enter too soon.

What I was going to add with 2023.9, by front end does occasionally seem to bog down (is the best way I can decribe it). It'll work fine, then suddenly get laggy to the point of not usable. But if I close or refresh the browser tab it seems to come good. But annoyingly it doesnt do it all the time, so I have yet to work out what might trigger it. It should be noted that sometimes I just open the HA dashboard, and then go straight to the settings menu, but it will still slow down. So it doesnt seem to be just the user editable lovelace part of the dashboard.

cpuks commented 9 months ago

I checked all those things - no device tracker in my setup so no file, also I'm running the stock theme black version so that's not the case too. Just installed a different browser than I use day-to-day - same behaviour on Chrome - I go to devices -> MQTT for example -> use filter once and it's ok then try to filter for different entities and it's extremely slow - as mentioned above - faster to kill tab and re-open.

joostlek commented 9 months ago

Just discussed this with someone from frontend, can you try logging with system log on debug level? That might get us more information

logger:
  logs:
    homeassistant.components.system_log: debug

joostlek commented 9 months ago

Also to make the picture complete, are you using any custom cards or frontend stuff? And although you also tried on a different browser, I just want to try to exclude everything at this point, any browser extensions?

cpuks commented 9 months ago

Give me a day or two please as I need to sort out higher matters (wife kids etc.) - just added the logger part and I'll restart HA soon. Also some hacs addons but to be honest I've been using the same things as before for months. Some of them are: browser_mod, dwains dashboard, etc - but what I have as default is the auto-populated that comes stock - I just came to the point where everything is done without any UI digging.

Anyway, list of any of the integrations installed via HACS attached.

config_entry-hacs-b2532ce1364a3b0c7dc57e939cfbf72d.json.txt

jerome6994 commented 9 months ago

Good evening

So I can't find the origin of the slowdown! But really no more worries at home, it’s really strange! I can't reproduce the phenomenon on Chrome or Firefox!

There is still one problem though! This is at the level of the MQTT integration which does not restart properly.

This is the last topic I have left since the September update

cpuks commented 9 months ago

I've been running 2023.9.3 for a few days now, but the issue still persists. @joostlek attached logs with homeassistant.components.system_log: debug
home-assistant.log

I can easily reproduce this problem by: 1) start fresh HA 2) go to integrations 3) enter MQTT or esphome (doesn't matter in the end as once I do those steps UX starts to slow down noticeably - it can be any integration) 4) pick one of the devices - check the entity details 5) go back to integrations again 6) enter MQTT or esphome again and now it's starting to slow down and it's getting worse and worse until I close the browser and start again

You can find examples below - notice the slowdown when entering an entity to display its history/states As I mentioned it's VM with 2 cores of i5-1240p and 4 GB RAM running on NVMe, recorder is set to 7 days, postgresql running outside HA (same proxmox machine) inside docker container, long term stats are inside influxdb running in a separate docker container outside HA <- CPU utilization during those screen recordings was between 1-1.5% - no spikes or any other delays. mqtt.webm esphome.webm

For screen recording I've used WIn 11 laptop and Edge browser, I've got same behaviour on mobile, and on iPad with Safari too

jerome6994 commented 9 months ago

Good morning

I have not found the reason for the resolution of my slowdowns but I confirm that the subject is completely resolved. No problems since my last information.

Shouldn't we open a new issue because the subjects may be different ?

Jérôme

deanfourie1 commented 9 months ago

Not sure about you guys, but this is now really bad for me.

Even in desktop web browser, its basicly unusable becoming so laggy.

cpuks commented 9 months ago

Not too much input but issue exists for sure. @joostlek did you have a look at logs I've attached?

joostlek commented 9 months ago

Euh not yet, I was a bit all over the place last week, I'll have a look in a couple of minutes

joostlek commented 9 months ago

I just checked them, and I did not see what I was expecting. I was expecting frontend logs. (unless you havent visited the dashboard during the logs, but I find that hard to believe). I will try to get to know what to do to get more logs, and hopefully more info about this issue

cpuks commented 9 months ago

@joostlek OK, in the meantime I've upgraded to 2023.10.0 and the issue persists - it's really bad, to be honest - I can reproduce the problem just by going back and forth between Settings->Devices->Integrations a couple of steps there and it's extremely slow while CPU usage is unchanged. Did you check the screen recording I've provided as it clearly shows issue Something is odd with mem usage - HA reports 1.9 GB / 4 GB avail - and proxmox says 3.64 GB / 4 GB - as far as I remember it was always identical.

cpuks commented 9 months ago

I've opened new issue as it seems it's frontend problem https://github.com/home-assistant/frontend/issues/18127

steverep commented 9 months ago

Is there anyone experiencing this problem not using proxmox?

This seems like a connection issue of some sort rather than anything that can be fixed in the frontend, but go ahead and try:

Clear all site data for your domain in the browser (not just cache). For example in Firefox this would be under Settings > Cookies & Site Data > Manage Data. Should be ~20 MB unless something is off.
Log in and try again.
If problem persists, close browser and login again, wait 10 seconds or so then set your browser to work offline. Test for slowness again (realizing that things like entity states and such won't be available).

For those who might be adept with browser dev tools, please look for clues regarding the web socket connection in the network tab.

cpuks commented 9 months ago

I just installed fresh Firefox - so as per point 1&2 - less than 60 sec and it's super unresponsive - check the attached screen recording screen-recorder-thu-oct-05-2023-21-46-06.webm

This is CPU/MEM usage from HA during that screen recording: This is data from proxmox:

System is idle to be honest but UI is almost unusable after a minute. The only wierd thing - why HA does not use more RAM? Can anyone check % of RAM usage?

Did the same on a different PC with fresh chrome while on 2023.9 so it's not browser-specific or OS-specific as the same applies to Android and iOS

steverep commented 9 months ago

By "fresh" you mean had never been installed before? Otherwise that would not guarantee zero site data.

Also you stopped at step 2...

home-assistant / core