Ali8904
commented
1 year ago Got the exact same error while installing on a raspberry pi 3
In case someone has solved it, dunklerphoenix and me would be very appreciative.
Closed DunklerPhoenix closed 10 months ago
Ali8904
commented
1 year ago Got the exact same error while installing on a raspberry pi 3
In case someone has solved it, dunklerphoenix and me would be very appreciative.
agners
commented
1 year ago @DunklerPhoenix @Ali8904 are those new installations?
The network configurations are stored on a separate overlay partition. It seems that this overlay partition is full.
Can you connect an HDMI screen and keyboard and login to the OS shell using login, then run the following commands:
df -h /mnt/overlay/
du -sh /mnt/overlay/*Instead of using screen/keyboard you can also use the OS debug SSH access on port 22222 (see this guide how-to setup SSH access through port 22222).
Ali8904
commented
1 year ago Not a new installation, just want to update the system. The system was up to date, so there was no big jump.
Usually I'm logged in via the ip address using a browser. I'll try that solution.
Note: there is way more luck involved than knowledge of it-topics 😅
DunklerPhoenix
commented
1 year ago The informations from the commands are maybe not so informative. I tried to access via 22222 (didn't work. Port stays closed) and restarted HassOS multiple times. So at the moment was the internet connection detected (maybe I try it later), but the error message above is vor /dev/Networkmanager and so I made the df -h for it.
Unfortunately I forgot the du -sh for /etc/NetworkManager. I don't have access to the HassOS at the moment so I will try to deliver it later
If you have any other commands that I should run then say it please
Ali8904
commented
1 year ago well even though, i do not understand that much when it comes to IT, but i would say, that doesnt work. Or: (high probability) i missed something.
Edit: after a bit of research, i came to the tip to restart HA via System/Hardware. up until now, i restartet it just under seting full restart (dozents of times). after restarting it that way, the supervisior update was no longer available to update. sooo, i guess that issue is gone - somehow
agners
commented
1 year ago @DunklerPhoenix that information is already helpful.
It seems there is space on the overlay parititon, so not sure where this is comming from :thinking:
If you have any other commands that I should run then say it please
/etc/NetworkManager is still on the read-only rootfs partition, so Available 0 is expected there. But the subdirectory system-connections should have space. Can you check:
df -h /etc/NetworkManager/system-connections/well even though, i do not understand that much when it comes to IT, but i would say, that doesnt work. Or: (high probability) i missed something.
You can't use the SSH add-on as it is in a containerized environment. You need to use Keyboard/Mouse or setup the debug SSH on port 22222 as described in https://github.com/home-assistant/operating-system/issues/2752.
Ali8904
commented
1 year ago Okay, just eddited my coment, while you did reply (thank you for that)
but what is different, using a display and a Keyboard, instead of exxising via the IP and using a Terminal on HA? Just to learn a bit more
DunklerPhoenix
commented
1 year ago Yes it have space left
Edit: but still no space left on saving the network settings
rexbron
commented
1 year ago I am having the same issue with not being able to add network interfaces via GUI, HA command or via NMCLI.
Home Assistant 2023.8.4 Supervisor 2023.08.1 Operating System 10.5 Frontend 20230802.1 - latest
Running on RasPi4
I have logged in via ssh debugging:
@agners @bdraco I think /mnt/overlay/var/lib/bluetooth/DC:A6:32:65:DC:FE/cache/ is the culprit. I live close to a busy road and it's picking up all the bluetooth devices of people and cars passing by.
Deleting a few of the files allows my ha net vlan command to succeed.
# df -h /mnt/overlay/
Filesystem Size Used Available Use% Mounted on
/dev/mmcblk0p7 87.5M 25.1M 55.6M 31% /mnt/overlay
# du -sh /mnt/overlay/*
20.0K /mnt/overlay/etc
12.0K /mnt/overlay/lost+found
4.0K /mnt/overlay/root
25.1M /mnt/overlay/var
# df -h /etc/NetworkManager/system-connections/
Filesystem Size Used Available Use% Mounted on
/dev/mmcblk0p7 87.5M 25.1M 55.6M 31% /mnt/overlayOf note, I have no more inodes available:
# df -i /mnt/overlay
Filesystem Inodes Used Available Use% Mounted on
/dev/mmcblk0p7 24576 24576 0 100% /mnt/overlaySuggested from this serverfault post
Following a blog post about what to do when you run out of inodes
# du -d 3 /mnt/overlay | sort -n | tail
6 /mnt/overlay/etc/NetworkManager/system-connections
7 /mnt/overlay/etc/NetworkManager
9 /mnt/overlay/var/lib/NetworkManager
12 /mnt/overlay/lost+found
20 /mnt/overlay/etc
166 /mnt/overlay/var/lib/systemd
25523 /mnt/overlay/var/lib/bluetooth
25699 /mnt/overlay/var/lib
25700 /mnt/overlay/var
25737 /mnt/overlay
# ls -1 /mnt/overlay/var/lib/bluetooth/DC:A6:32:65:DC:FE/cache/ | wc -l
24505Okay, just eddited my coment, while you did reply (thank you for that)
but what is different, using a display and a Keyboard, instead of exxising via the IP and using a Terminal on HA? Just to learn a bit more
The SSH add-on accesses the home assistant container only.
Access to the host system outside of the docker container. By default, you can't SSH into the host system without adding ssh keys as described by the debugging HAOS docs.
rexbron
commented
1 year ago @DunklerPhoenix could you post the results of :
df -i /mnt/overlay
du -d 3 /mnt/overlay | sort -n | tail
to see if you also have no inodes?
DunklerPhoenix
commented
1 year ago Nope. No inodes left
agners
commented
1 year ago @rexbron @DunklerPhoenix very interesting findings, thanks for this investigations.
I checked some of my test devices, and the used inode count is very low. While I live in a multi-tenant building, my location doesn't seem to pick up Bluetooth devices from outside.
So both of you have 1000's of cache entries in the Bluetooth cache directory?
I guess increasing the inode count would be one way to work around the problem. However, a cleaner fix would be to introduce better cache management, e.g. delete devices from the Bluetooth cache every now and then :thinking:
DunklerPhoenix
commented
1 year ago Sounds logical. I live above a pub. So there are many people around that get caught by the Bluetooth dongle. And I saw in the logs many BT Errors with no space left yesterday.
I would never thought of bluetooth as the source of the problem xD
bdraco
commented
1 year ago Can someone with the issue send a profiler callgrind file using the profiler.start service?
https://www.home-assistant.io/integrations/profiler/
The file only contains the names of the python code function calls and names of the libraries installed so it should be safe to post here unless you don't want people to know which integrations you have installed
rexbron
commented
1 year ago @bdraco I have attached the callgrind.out as zip (to allow for uploading to the github comment). Let me know if this meets your requirements.
DunklerPhoenix
commented
1 year ago callgrind.out.1694691960524360.zip
Have fun looking at this mess xD
bdraco
commented
1 year ago @bdraco I have attached the callgrind.out as zip (to allow for uploading to the github comment). Let me know if this meets your requirements.
Thanks @rexbron . Looks good. I don't see a massive amount of bluetooth traffic in the profile, but it might be that bluetooth has failed due to the issue. I suspect https://github.com/home-assistant/operating-system/pull/2751 will fix the issue for you. It would be good to get a new profile to check everything again once you have installed an OS build with 2751 in it.
bdraco
commented
1 year ago callgrind.out.1694691960524360.zip
Have fun looking at this mess xD
@DunklerPhoenix Your profile is much more interesting with 3719 bluetooth advertisements per minute. This will be faster in 2023.10.x as we recently fixed some performance bottlenecks there. It makes sense why you see this issue. I expect https://github.com/home-assistant/operating-system/pull/2751 will be a good solution here.
On a side note: It also looks like the mqtt_room integration is not well optimized and is responsible for the bulk of your run time. Specifically this function https://github.com/home-assistant/core/blob/a62f16b4cc7a20fe6162d8ad858aa08423466e19/homeassistant/components/mqtt_room/sensor.py#L130
rexbron
commented
1 year ago @DunklerPhoenix You can run the command below from the physical keyboard to manually clear the cache
find /var/lib/bluetooth/*/cache -mindepth 1 -type f -atime +7 -exec rm {} \;
rexbron
commented
1 year ago @bdraco callgrind.out.1694785421594926.zip Ran the cache clearing command yesterday and re-ran the profiler this morning
DunklerPhoenix
commented
1 year ago find /var/lib/bluetooth/*/cache -mindepth 1 -type f -atime +7 -exec rm {} \;
Hahahah the command took a really long time to finish. Nearly one minute xD
Btw it worked. I can save my network settings again <3
bdraco
commented
1 year ago @bdraco callgrind.out.1694785421594926.zip Ran the cache clearing command yesterday and re-ran the profiler this morning
@rexbron Thanks for that. Bluetooth looks like its working correctly now. Your profile looks very clean but there is some I/O in your event loop. I fixed it in https://github.com/home-assistant/core/pull/100455.
github-actions[bot]
commented
10 months ago There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
rexbron
commented
10 months ago @DunklerPhoenix This should be marked as closed with the release of HAOS
agners
commented
10 months ago It is part of HAOS 11.0 and newer (see https://github.com/home-assistant/operating-system/releases/tag/11.0.rc1 changelog)
Describe the issue you are experiencing
Heho Home Assistant often can't install updates, addons or anything else anymore because it says "no host internet connection" I tried some solutions like changing the DNS Server, but I can't save the settings. If I try i get the message:
Can't update config on eth0: failed to update connection: error writing to file '/etc/NetworkManager/system-connections/Supervisor eth0.nmconnection': failed to create file /etc/NetworkManager/system-connections/Supervisor eth0.nmconnection.ZNJC91: No space left on deviceEDIT: Hardware is HA Blue
Do you know what this can be? After a system restart the host internet connection is for some time available again
What type of installation are you running?
Home Assistant OS
Which operating system are you running on?
Home Assistant Operating System
Steps to reproduce the issue
1. 2. 3. ...
Anything in the Supervisor logs that might be useful for us?
System Health information
System Information
Home Assistant Community Store
GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4971 Installed Version | 1.32.1 Stage | running Available Repositories | 1330 Downloaded Repositories | 85 HACS Data | okHome Assistant Cloud
logged_in | true -- | -- subscription_expiration | 28. August 2023 um 02:00 relayer_connected | true relayer_region | eu-central-1 remote_enabled | true remote_connected | true alexa_enabled | false google_enabled | false remote_server | eu-central-1-2.ui.nabu.casa certificate_status | ready can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | okHome Assistant Supervisor
host_os | Home Assistant OS 10.4 -- | -- update_channel | stable supervisor_version | supervisor-2023.08.1 agent_version | 1.5.1 docker_version | 23.0.6 disk_total | 113.9 GB disk_used | 21.4 GB healthy | true supported | true board | odroid-n2 supervisor_api | ok version_api | ok installed_addons | FTP (4.7.1), File editor (5.6.0), Advanced SSH & Web Terminal (15.0.6), Node-RED (14.4.4), MariaDB (2.6.1), RaspberryMatic CCU (3.69.7.20230626), AppDaemon (0.13.4), Log Viewer (0.15.1), ledfx (d7e3146), motionEye (0.19.1), AdGuard Home (4.8.14), WireGuard (0.8.1), chrony (2.6.0), Rhasspy Assistant (2.5.11), Traefik (4.2.0), Z-Wave JS (0.1.85), Bitwarden secrets for Home Assistant (1.4.6), Home Panel (2.6.1), Netdata (2.1.1), UniFi Network Application (2.7.0), ESPHome (2023.7.1), Dropback (7.1.1), heimdall (0.3.1), Piper (1.2.0), Whisper (1.0.0), Bookstack (1.9.1), Mosquitto broker (6.2.1), Silicon Labs Multiprotocol (2.3.1), Matter Server (4.9.0), Rclone Backup (3.0.1)Dashboards
dashboards | 3 -- | -- resources | 61 views | 15 mode | storageRecorder
oldest_recorder_run | 2. August 2023 um 22:58 -- | -- current_recorder_run | 9. August 2023 um 15:02 estimated_db_size | 1743.35 MiB database_engine | sqlite database_version | 3.41.2Sonoff
version | 3.5.2 (2ad1cd7) -- | -- cloud_online | 1 / 3 local_online | 1 / 1Supervisor diagnostics
config_entry-hassio-1dd8512d8dc2f72a08934be72bd4720b.json.txt
Additional information
No response