search

homebridge / homebridge-config-ui-x

The Homebridge UI. Monitor, configure and backup Homebridge from a browser.
https://homebridge.io
MIT License
2.59k stars 369 forks source link

[Homebridge UI] Restore backup failed: zlib: unexpected end of file when trying to restore a backup on new HB instance on new machine. #1856

Open buddyrich33 opened 7 months ago

buddyrich33 commented 7 months ago

Current Situation

Trying to move my homebridge instance from one Mac to a new one and I am getting this error. [11/26/2023, 10:46:17 AM] [Homebridge UI] Restore backup failed: zlib: unexpected end of file

The homebridge I am trying to back up is a Mac Mini 2018 (so intel CPU) running latest MacOS Sonoma. homebridge is 1.7.0 plugins and Node version are up to date (node version 20.10.0)... I can make and download my backup file fine. I stop my hb instance via hb service command,.

The new mac I am trying to restore to is an M2 Max Mac Studio, running 1.7 homebridge and same 20.10.0 node version. No plugins yet.. I believe the restore will install them.

The backup file itself is 1.7 MBs and unzips with no errors and I can see the config files.. so it doesn't seem to be a corrupt backup.

Logs

Even with debug logging on this is the only error I get in the logs:

[11/26/2023, 10:46:17 AM] [Homebridge UI] Restore backup failed: zlib: unexpected end of file

Config

No response

Homebridge UI Version

4.53.0

Homebridge Version

1.7.0

Node.js Version

20.10.0

Operating System

macOS

Environment Info

Raspberry Pi Model

None

bwp91 commented 7 months ago

Hi @buddyrich33 is your version of Homebridge UI up to date? The latest (at time of my comment) is v4.53.0, do you get this error message about restoring on this version of UI?

buddyrich33 commented 7 months ago

Yes. It was 4.53.0... Though I solved my issue, though I don't know which file was the culprit.

I thought perhaps my backup file was too large, at around 1.6MB...

I unziped the backup file and saw the expected configuration jsons, etc... in the storage folder I noticed two plugins in particular also had gziped logs... hue plugin, had maybe a 100KB or so so log, but the hubitat maker-api had multipart logs, with the active one almost 7MB in size. There was also a strange zero-byte file called _persist.json... Anyway I deleted the logs and the zerobyte file, re gziped them up and was able to use the much smaller backup file to restore successfully.

sapireli commented 7 months ago

This is a recurring issue with the latest version. The error is the same always: "Restore backup failed: zlib: unexpected end of file"

Testing was done on a raspi running debian bookworm image of dietpi.

The backup comes in at 6.3mb compressed.

Upon closer inspection: there is a large hidden ".npm" folder inside the backup. This folder is a relatively large sub-folder -> _cacache/content-v2/sha512/ -> which contains over 100 two letter folders. Removing this folder and gzipping allows the backup file to be successfuly uploaded but then the terminal window reports the following error: "Error: Uploaded file is not a valid Homebridge Backup Archive." "

NorthernMan54 commented 7 months ago

@sapireli - These are the conditions that trigger that message - https://github.com/homebridge/homebridge-config-ui-x/blob/f63405f68a55416be3f9bb3ee4d47227b78d691c/src/modules/backup/backup.service.ts#L407

Your rebuilt zip file must be missing one of them

PS - Is it possible to share the broken backup file to me, so we can look at remediating the issue ? I'm on the Homebridge discord server so we could leverage that.

sapireli commented 7 months ago

@sapireli - These are the conditions that trigger that message -

https://github.com/homebridge/homebridge-config-ui-x/blob/f63405f68a55416be3f9bb3ee4d47227b78d691c/src/modules/backup/backup.service.ts#L407

Your rebuilt zip file must be missing one of them

PS - Is it possible to share the broken backup file to me, so we can look at remediating the issue ? I'm on the Homebridge discord server so we could leverage that.

Thank you so much for pointing me there, i submitted the fix in #1948 its working well now.

StefanGebhardt commented 6 months ago

Bug is still there in 4.54.2-beta.58 Updated to this version on the source installation, made a backup, tried to restore in the new installed installation (also 4.54.2-beta.58). Still the same error...

bwp91 commented 6 months ago

Hi @StefanGebhardt

assuming you are still on the beta version, can you generate and download a new backup file, download and locate it in your finder/windows explorer and see how large the file is?

StefanGebhardt commented 6 months ago

It has 15 MB (14.841.218 bytes)

bwp91 commented 6 months ago

Are you able to extract it on your local file system?

StefanGebhardt commented 6 months ago

Yes, with no problems on Mac OS. Need any information about the files inside?

bwp91 commented 6 months ago

Just knowing what is generally taking up the most space would be useful.

Iā€™m unsure of what a typical backup would be in terms of size, but 15MB seems a lot for just some text files.

Perhaps there are more directories we need to ignore. and perhaps this error is showing when the file is simply too large šŸ§

StefanGebhardt commented 6 months ago

These are the files & directories after extracting:

Untitled

StefanGebhardt commented 6 months ago

I'll tried a lot of deleting and compacting again. Especially the big *persist.json-Files. Nothing works... :-(

bwp91 commented 6 months ago

@StefanGebhardt are you able to provide another screenshot with those top folders expanded to see which files in them are taking up the most space?

I've done a bit of testing locally, and it seems that whilst uncompressing, if the new folder gets over ~1.2mb then this error is thrown.

bwp91 commented 6 months ago

Hold that above thought above

@StefanGebhardt @buddyrich33 @sapireli

Are you able to test a restore process on the beta version of UI? At least v4.54.2-beta.78 ?

sapireli commented 6 months ago

i just tested backup and restore on the 4.54.2-beta.79, and it is working well.

@StefanGebhardt Please expand all the folders, and show hidden files as well. Something seems off.

sapireli commented 6 months ago

FWIW my backup of an extensive homebridge config clocks in at a heft weight of 125 KBs on the beta.

sapireli commented 6 months ago

@StefanGebhardt also try running "npm cache clean --force" on your homebridge server, then run the backup again see if the backup size gets smaller...

bwp91 commented 6 months ago

@StefanGebhardt no need do any of this

bwp91 commented 6 months ago

sorry @sapireli didnt mean to overrule you there but there indeed was a an issue with the UI, so any backups should be restore-able now

StefanGebhardt commented 6 months ago

@bwp91 Updated to Beta.79 and it worked without any problems. Thank you so much!

github-actions[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 5 months ago

This issue has been closed as no further activity has occurred.

Goldsucher commented 3 months ago

I still have this error with latest version of homebridge-ui. My backup file is 44.4 MB. There are many .zigbee files which blow up the file (I use homebridge-deconz) Do I need these .zigbee files? I tried to remove them and compress the folder with Keka (Mac App) but then I get the error "Error: Uploaded file is not a valid Homebridge Backup Archive."

Could someone help me with this?

EDIT: Okay I deletted the files, made the backup und after the restore everything still works šŸ‘

NorthernMan54 commented 3 weeks ago

Was working with a user and identified that restoring a large homebridge file works on 4.50.6. We would need to determine what changed between 4.50.6 and the current release.

NorthernMan54 commented 3 weeks ago

When a restore occurs, 1048576 bytes are passed to src/modules/backup/backup.controller.ts but the file is 1107901 bytes, triggering the error.