search

thomasmauerer / hassio-addons

My collection of addons for Home Assistant
MIT License
458 stars 54 forks source link

Backups with size of ~ 4 GB not copied to remote SAMBA share (SAMBA backup fails) #161

Open bcutter opened 10 months ago

bcutter commented 10 months ago

SAMBA backup fails for backups of roughly around or greater than 4 GB. Everything below that mark works just fine. For a weekly backup which now hit the 4 GB size mark, it is not uploaded to the remote SAMBA storage.

grafik

Of course this can be highly dangerous if the local storage dies - and the important backup is not available remotely.

Logs of latest occurence:

[23-11-11 02:39:36] INFO: Backup finished
[23-11-11 03:30:00] DEBUG: Input received: {"command":"trigger","backup_name":"{type} Backup nur InfluxDB {version} {date}","exclude_folders":["addons/local","media","ssl","share","homeassistant"],"exclude_addons":["xxxMASKEDxxx"]}
[23-11-11 03:30:01] INFO: Running backup with customized parameters
[23-11-11 03:30:01] INFO: Backup running ...
[23-11-11 03:30:02] DEBUG: Backups local/remote: 3/45
[23-11-11 03:30:02] DEBUG: Total backups succeeded/failed: 937/166
[23-11-11 03:30:02] DEBUG: Last backup: 2023-11-11 02:39
[23-11-11 03:30:02] DEBUG: Posting sensor data to API at /core/api/states/sensor.samba_backup
[23-11-11 03:30:03] DEBUG: API Status: 200
[23-11-11 03:30:03] DEBUG: API Response: {"entity_id":"sensor.samba_backup","state":"RUNNING","attributes":{"friendly_name":"Samba Backup","backups_local":"3","backups_remote":"45","total_backups_succeeded":"937","total_backups_failed":"166","last_backup":"2023-11-11 02:39"},"last_changed":"2023-11-11T02:30:03.102345+00:00","last_updated":"2023-11-11T02:30:03.102345+00:00","context":{"id":"01HEY3E02YMRQ66JES9HNTG9FF","parent_id":null,"user_id":"xxxMASKEDxxx"}}
[23-11-11 03:30:03] INFO: Creating backup "Partial Backup nur InfluxDB 2023.4.6 2023-11-11 03:30"
parse error: Invalid numeric literal at line 1, column 5
[23-11-11 06:31:42] DEBUG: Posting sensor data to API at /core/api/states/sensor.samba_backup
[23-11-11 06:31:42] DEBUG: API Status: 200
[23-11-11 06:31:42] DEBUG: API Response: {"entity_id":"sensor.samba_backup","state":"FAILED","attributes":{"friendly_name":"Samba Backup","backups_local":"3","backups_remote":"45","total_backups_succeeded":"937","total_backups_failed":"167","last_backup":"2023-11-11 02:39"},"last_changed":"2023-11-11T05:31:42.672672+00:00","last_updated":"2023-11-11T05:31:42.672672+00:00","context":{"id":"01HEYDTM6GPDF78HQQ74KZJET5","parent_id":null,"user_id":"xxxMASKEDxxx"}}
[23-11-11 06:32:26] DEBUG: Posting sensor data to API at /core/api/states/sensor.samba_backup
[23-11-11 06:32:26] DEBUG: API Status: 200
[23-11-11 06:32:26] DEBUG: API Response: {"entity_id":"sensor.samba_backup","state":"IDLE","attributes":{"friendly_name":"Samba Backup","backups_local":"3","backups_remote":"45","total_backups_succeeded":"937","total_backups_failed":"167","last_backup":"2023-11-11 02:39"},"last_changed":"2023-11-11T05:32:26.867523+00:00","last_updated":"2023-11-11T05:32:26.867523+00:00","context":{"id":"01HEYDVZBKGG3SCB18C1M45HNX","parent_id":null,"user_id":"xxxMASKEDxxx"}}
[23-11-11 06:32:27] INFO: Backup finished
[23-11-12 02:00:53] INFO: Backup running ...

Possibly similar to https://github.com/thomasmauerer/hassio-addons/issues/151

Need assistance on this.

Mikescotland commented 10 months ago

I had a similar problem - backup is creating first a tmp backup and then writes it to Samba. In my case, HA drive became full and backup continued - after a few hours stopped. Interesting thing is that the drive still had 40 GB free, I noticed a notification in Ubuntu about the drive being full. Not sure what happened though - the media folder is around 6GB, I excluded it now and works. Looks like the addon does something in this case that fills up the drive (backup was running for.... 3hr before failed, unlikely on a core i5 machine to take so long)

bcutter commented 10 months ago

I can't follow you completely. Did you have storage issues on the source (HA) or the target (remote SAMBA share)? As mentioned the backup is created fine locally (where storage could be an issue in theory, but is not, see [*1], but not transferred to the remote storage (which has more than enough space by the way).

[*1] grafik

Mikescotland commented 10 months ago

Source. Despite having around 50GB free and media folder only 8GB, Ubuntu threw and error after an hour of backup saying drive is completely full. Tmp file on source drive was massive, like backup was endlessly adding and adding..

bcutter commented 10 months ago

OK, thanks for your report (which made me check storage, didn't think about that before). Very likely not the case (not a storage issue) here.

ouaibe commented 9 months ago

This seems very similar to #65 and #137 and #150 currently not solved until #79 implementation.

bcutter commented 9 months ago

...which is open for almost 2 years. Wow. Addon slowly dying... no big hopes. So sad.

bcutter commented 5 months ago

Intermediate informatione:

For me some supervisor update seems to have boosted backup creation by a lot already somewhere in December 2023. Problem likely never was/is not the actual backup size, but the time needed to create it. Because the latter one decreased dramatically, this issue did not happen anymore for me - currently. Anyway, this addon seems to run into some limitations (which will likely arise again once the backup size and therefore duration increases again over time).

ouaibe commented 2 months ago

Re-enabled the extension on my end and it seems the time to take backups has significantly improved and thus the timeout doesn't happen anymore, 'fixing' this issue. My backup files are > 4GB.

bcutter commented 2 months ago

Re-enabled the extension on my end and it seems the time to take backups has significantly improved and thus the timeout doesn't happen anymore, 'fixing' this issue. My backup files are > 4GB.

So you confirmed https://github.com/thomasmauerer/hassio-addons/issues/161#issuecomment-2052445744.

ouaibe commented 1 month ago

Small update: after 2 month running weekly, the backups are working fine, including file rotation. My backup files are about 7GB and I haven't seen this issue re-appear. I would conclude this issue can be closed as "solved".

bcutter commented 1 month ago

My larger ones are roughly 6.x GB (only once a week) and are copied fine to the remote SAMBA share too. I think the problem never was the size of the backup, but instead the backup creation duration. See https://github.com/thomasmauerer/hassio-addons/issues/65#issuecomment-1509727085 or https://github.com/thomasmauerer/hassio-addons/issues/161#issuecomment-2052445744.

As we still don't know if it's because of the HA internal backup improvements (Supervisor? Core?) etc. and the actual core issue https://github.com/thomasmauerer/hassio-addons/issues/65 still is unsolved), I'll leave this open until the maintainer decides what to do with this issue. Because having such issues open for 9 months without a single reaction is... negatively totally awesome :-)

I think once the backup creation would (will! a matter of time...) exceed 3 hours, everything will be reset to start - so from a technical perspective, the issue in this addon is still not fixed. It has just been mitigated by significant performance improvements of the HA Supervisor.