Downloaded backup size is much larger than content size.

[niemyjski]

The problem

I noticed when downloading a snapshot (created via snapshot service) via edge latest on Windows 11 that the file size is much larger than the content size. I'm not sure what could be causing this but it could be something to do with the streaming

What version of Home Assistant Core has the issue?

2021.10.6

What was the last working version of Home Assistant Core?

2021.08.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Snapshots

Link to integration documentation on our website

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[mib1185]

Are you sure, that the downloaded file is ~45GB 🤔

[niemyjski]

Yes

[mib1185]

Please try the backup_full service, since snapshot_full is depreciated

[probot-home-assistant[bot]]

hassio documentation hassio source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @home-assistant/supervisor, mind taking a look at this issue as it has been labeled with an integration (hassio) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[ludeeus]

This sounds like a bug in Windows, or even 7zip. What the Supervisor report is what's correct.

[mib1185]

direct downloaded via supervisor UI (_created by backup_full service_) : size compared between supervisor and windows and 7zip ✔️

[niemyjski]

I'm running a hassio.backup_full backup now and it takes forever so I'll follow up when that is done. I was using the hacs addon due to auto purge, it's a proxy and calls the backup apis (confirmed from looking at the source). My apologies, I had the terminology wrong... I knew it was either backups or snapshots and one was deprecated.

[niemyjski]

I'll have to check it in the morning the backup is taking way too long on a beast of a machine with only NVME storage..

21-10-25 20:20:02 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_influxdb
21-10-25 21:12:34 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_influxdb

[niemyjski]

Definitely a bug of some kind here. I ran one full backup using the service call at 8:20pm:

And then a daily at midnight using the same service call:

[niemyjski]

The default service call says it will have a date based file name, but when It's created it doesn't seem like it and the downloaded file is called tar, this probably should be it's own issue.

Regardless, there seems to be some differences between the backup api that that addon is calling and this service call.

tar is the backup from 8:20pm

tar(1) is the backup from midnight

One has a media compressed tar and the other doesn't (at least with content), Not sure why both don't have it but seems like part of the core issue. Perhaps it's being removed (bug?) but the compressed file isn't shrunk?

[mib1185]

could you please share the corresponding logs from supervisor for both backup jobs?

[niemyjski]

Is there a good way to view the supervisor logs? They seem truncated and I can only see the logs for the last 3 hours (no restarts have happened).

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

Still an issue with latest version

[agners]

@niemyjski logs of the two backup types (via addon and directly calling the service) should be helpful here.

You can get older Supervisor logs using journalctl on the OS shell. To get access to the OS shell, you have to follow this guide.

You then can get logs using:

journalctl -t hassos-supervisor

[niemyjski]

I'm no longer calling that addon. Is there any way to get these logs without enabling ssh and rebooting the server? I have a lot of services running on this that I'd prefer not to roll the whole box multiple times. Can this be exposed as part of the supervisor logs or maybe a feature to download a support zip that contains this and other logs?

Here are the logs from the supervisor ui

21-12-29 09:44:12 INFO (MainThread) [supervisor.backups.manager] Found 4 backup files
21-12-29 09:44:21 INFO (MainThread) [supervisor.backups.manager] Creating new full backup with slug e5b1dd9f
21-12-29 09:44:21 INFO (MainThread) [supervisor.backups.manager] Backing up e5b1dd9f store Add-ons
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_ada
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_ada
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_samba
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_samba
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_lutron-cert
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_lutron-cert
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_git_pull
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_git_pull
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_aircast
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_aircast
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_logviewer
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_logviewer
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_grafana
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_grafana
21-12-29 09:44:21 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_almond
21-12-29 09:44:22 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_almond
21-12-29 09:44:22 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_influxdb
21-12-29 10:49:10 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_influxdb
21-12-29 10:49:10 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_vscode
21-12-29 10:49:38 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_vscode
21-12-29 10:49:38 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_mariadb
21-12-29 11:20:48 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_mariadb
21-12-29 11:20:48 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_esphome
21-12-29 11:20:48 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_esphome
21-12-29 11:20:48 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on 03cabcc9_ring_mqtt
21-12-29 11:20:48 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon 03cabcc9_ring_mqtt
21-12-29 11:20:54 INFO (SyncWorker_2) [supervisor.docker.addon] Export image ccab4aaf/amd64-addon-frigate to /data/tmp/tmp0zdlezrd/image.tar
21-12-29 11:20:56 INFO (SyncWorker_2) [supervisor.docker.addon] Export image ccab4aaf/amd64-addon-frigate done
21-12-29 11:20:56 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on ccab4aaf_frigate
21-12-29 11:23:24 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon ccab4aaf_frigate
21-12-29 11:23:24 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_zwave_js
21-12-29 11:23:25 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_zwave_js
21-12-29 11:23:25 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on core_mosquitto
21-12-29 11:23:26 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon core_mosquitto
21-12-29 11:23:26 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on a0d7b954_ssh
21-12-29 11:23:26 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon a0d7b954_ssh
21-12-29 11:23:47 INFO (SyncWorker_5) [supervisor.docker.addon] Export image c7657554/amd64-addon-compreface to /data/tmp/tmp58_n6gpr/image.tar
21-12-29 11:23:53 INFO (SyncWorker_5) [supervisor.docker.addon] Export image c7657554/amd64-addon-compreface done
21-12-29 11:23:53 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on c7657554_compreface
21-12-29 11:31:35 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon c7657554_compreface
21-12-29 11:31:41 INFO (SyncWorker_2) [supervisor.docker.addon] Export image c7657554/amd64-addon-double-take to /data/tmp/tmpji1qgtki/image.tar
21-12-29 11:31:42 INFO (SyncWorker_2) [supervisor.docker.addon] Export image c7657554/amd64-addon-double-take done
21-12-29 11:31:42 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on c7657554_double-take
21-12-29 11:33:56 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon c7657554_double-take
21-12-29 11:33:56 INFO (MainThread) [supervisor.backups.manager] Backing up e5b1dd9f store folders
21-12-29 11:33:56 INFO (SyncWorker_11) [supervisor.backups.backup] Backing up folder addons/local
21-12-29 11:33:56 INFO (SyncWorker_11) [supervisor.backups.backup] Backup folder addons/local done
21-12-29 11:33:56 INFO (SyncWorker_1) [supervisor.backups.backup] Backing up folder media
21-12-29 11:45:57 WARNING (SyncWorker_1) [supervisor.backups.backup] Can't backup folder media: [Errno 2] No such file or directory: '/data/media/frigate/clips/front_door-1640799091.748688-619hyp.jpg'
21-12-29 11:45:57 INFO (SyncWorker_10) [supervisor.backups.backup] Backing up folder share
21-12-29 11:45:57 INFO (SyncWorker_10) [supervisor.backups.backup] Backup folder share done
21-12-29 11:45:57 INFO (SyncWorker_0) [supervisor.backups.backup] Backing up folder homeassistant
21-12-29 11:46:06 INFO (SyncWorker_0) [supervisor.backups.backup] Backup folder homeassistant done
21-12-29 11:46:06 INFO (SyncWorker_15) [supervisor.backups.backup] Backing up folder ssl
21-12-29 11:46:06 INFO (SyncWorker_15) [supervisor.backups.backup] Backup folder ssl done
21-12-29 11:46:48 INFO (MainThread) [supervisor.backups.manager] Creating full backup with slug e5b1dd9f completed

I noticed if I don't back up the media folder, influx, maria, or compre addons the size of the backup is 843mb. I don't think doing a partial backup would cause the smaller size.. I'm going to try creating a backup of each of the above separately to see if I can narrow it down.

The Exadel CompreFace addon was 2gb to backup and it should be really small. Working on the others

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

@agners can it be possible to get these logs from the web ui?

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

can it be possible to get these logs from the web ui?

[agners]

The Supervisor log is available from the frontend too, however, it only shows the last 100 lines or so. You have been asking for older logs, currently the only way to get to older logs is through the console.

[niemyjski]

Can we make this configurable?

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

Still an issue

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

Still an issue

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

Still an issue

[Bagunda]

I have the same problem. The backup size increases by gigabytes every month. Now I remembered that I have Transmission installed for downloading Torrent files. And the additional disk is mounted to the folder where the Home Assistant files are located (/share/, /usr/share/hassio/share/). It turns out that this folder is included in the backup. It looks like you need to unmount this disk before making a backup. I would like to automate this...

[Bagunda]

I confirm. After unmounting the disk, the backup became normal size.