Snapshot upload not working

dennisbrouwer91 commented 4 years ago

The problem

I woud like to migrate my Hass installation to a new VM, so i made a snapshot, downloaded that one and now trying to restore it on the new VM. I can select the upload option, select the file, but after a while it gives back a 502 Bad Gateway error.

Environment

Latest and greatest version of HassOS OVA (4.13)

Operating environment (OS/Container/Supervised/Core): OS

Problem-relevant `configuration.yaml`

No configuration.yaml file yet as this is in the setup wizard

Traceback/Error logs

502 Bad Gateway

In core log :

ERROR (MainThred) [homeassistant.components.hassio.http] Client timeout error on API request snapshots/new/upload

Additional information

Backup is 1.34GB.

springstan commented 4 years ago

@ludeeus could you please move this to the supervisor repo? Thanks 😊

dennisbrouwer91 commented 4 years ago

Sorry for posting it in the wrong one @springstan

springstan commented 4 years ago

Sorry for posting it in the wrong one @springstan

No worries 😄

ludeeus commented 4 years ago

Frontend is probably more fitting @springstan 502 is expected, that means the core is restarting, give it some time @dennisbrouwer91

dennisbrouwer91 commented 4 years ago

@ludeeus : I've waited HOURSSS and it's still gaving bad gateway

velkrosmaak commented 4 years ago

I have this issue too. Tried importing a snapshot (from 112.4) onto a clean 116.2 install at the beginning of the wizard - 502 Bad Gateway. Created an account and completed the wizard, then tried uploading the same snapshot from the snapshot UI, and I get this in the log:

Client timeout error on API request snapshots/new/upload This is in a VM on ESXi on an interface with a known working interface.

Zoogara commented 4 years ago

I have this issue too. Tried importing a snapshot (from 112.4) onto a clean 116.2 install at the beginning of the wizard - 502 Bad Gateway. Created an account and completed the wizard, then tried uploading the same snapshot from the snapshot UI, and I get this in the log:

Exactly same result when try to move config to new hardware.

dennisbrouwer91 commented 4 years ago

How i fixed it : Finish the setup wizard, install the SMB plugin, copy the backup and then do a restore from the snapshots page. That works fine!

Hy3n4 commented 4 years ago

I belie it is caused by size of a backup file. It will probably timeout before image is uploaded.

I just been able to restore 22MB backup (basicaly just config). It will alow you to restore into "minimal" installation. And then upload larger backup using rsync, scp or smb share and restore it.

deftdawg commented 3 years ago

This needs to be reopened, it is certainly not fixed. [as of HA 0.117.2]

I spent the morning trying to restore snapshots on hassio 4.15, none of them are restorable... Received the same message "502 Bad Gateway" everytime w/ Client timeout error on API request snapshots/new/upload showing in the docker logs homeassistant output.

My first try was a 560MB snapshot of which Home Assistant config tarball took 278MB and Open Z-wave tarball took up 164MB of that space. Failed.

I switched to partial backup, dropped Open Z-wave out, ran recorder.purge w/ {"keep_days":"31", "repack":"true"} to shrink my home-assistant_v2.db from 12GB down to 1 GB and managed to produce a snapshot at 266MB. Failed.

The only scenario this seems to work is backing up and restoring an empty virgin configuration of Home Assistant. :\

Here are some suggestions:

[ ] test the code with a reasonably large home-assistant configuration / history database snapshot
[ ] if there are limitations to the size consider warning users that the snapshots produced will not be usable
[ ] handle the error on upload and error out more gracefully (maybe check the size on the client side prior to sending it)
[ ] offer a configuration option to drop historical data home-assistant_v2.db during the backup to reduce size

deftdawg commented 3 years ago

Painful workaround to get a ~400MB snapshot restored on a fresh HassOS install:

create a full snapshot; if you make a partial even if it contains HomeAssistant config supervisor will reject it 20-11-02 20:33:21 ERROR (MainThread) [supervisor.snapshots] 4a386787 is only a partial snapshot!
Connect a screen and keyboard to the raspberry pi and hit enter to get a login prompt
Login w/ root (no password), type login at the ha> prompt
dropbear -B -R start ssh to remote in root@homeassistant.local (repeat step 3 when you connect to get shell)
docker exec -i -t homeassistant scp me@myworkstation:Downloads/backup_snapshot.tar . (scp is broken on HassOS /usr/bin/dbclient missing; can't cp dropbear to /usr/bin/dbclient because squashfs / is RO)
mv /mnt/data/supervisor/homeassistant/*.tar /mnt/data/supervisor/backup/
ha snapshots reload
Shutdown your old home assistant instance (don't know if this is required)
ha snapshots restore $(ha snapshots | grep slug | cut -d: -f2)

Open another ssh connection, do step 3 and then run docker logs -f hassio_supervisor to watch restoration do it's thing; this can take a long time on an RPi4.

20-11-02 20:46:12 INFO (MainThread) [supervisor.snapshots] Full-Restore f941c8fd start
....
20-11-02 21:09:31 INFO (MainThread) [supervisor.snapshots] Full-Restore f941c8fd done

Done! 😄

dennisbrouwer91 commented 3 years ago

This is indeed still broken, just tried on a brand-new installation!

marijneken commented 3 years ago

Just doing this today (after the n-th time an SD card failed me) and finding out this is definitely still a problem. I got the Bad Gateway message on a 300MB restore with a fresh install of 4.15. Thank you @deftdawg, your instructions saved my day.

deftdawg commented 3 years ago

@ludeeus Can this be reopened or should we spin up a duplicate to track this issue?

ludeeus commented 3 years ago

No need, all logs/errors have been removed in 0.118

Kisty commented 3 years ago

Still having the same issue on Home Assistant 0.118.4 on a raspberry pi 3 running on raspberry pi OS using supervised install.

Guyohms commented 3 years ago

I have the same problem on a fresh install of the image hassos_rpi3-64-4.16.img.gz on a RPI 3B+

During onboarding or trying to upload through the snapshot page results in a Bad Gateway error.

deftdawg commented 3 years ago

Some good news, it looks like this commit (https://github.com/home-assistant/core/pull/43851) merged into 0.118.5 could maybe help with this issue.

ThePatricide commented 3 years ago

I have a 1.6GB snapshot (keep a lot of log files) and also cannot restore a snapshot. It says 'Snapshot too large'.

deftdawg commented 3 years ago

@bjschuit if the message was Uploaded snapshot is too large, we can trace that to

Frontend: https://github.com/home-assistant/frontend/blob/590cd8500d88f992338a98c51db0c62c89ff7d4a/src/data/hassio/snapshot.ts#L112 Which connects to the backend @ api/hassio/snapshots/new/upload Which I think ends up here (??): https://github.com/home-assistant/supervisor/blob/6f26536d97c61feeb47857f666b69244a12091a5/supervisor/api/snapshots.py#L187

The only thing I see in that method is the possibility that you could run out of temp disk space to unpack the snapshot... That's just a guess though

At least it's no longer timing out on the uploading part now, @ludeeus can you comment on what's causing the upload size error and what users might do to remedy it?

blerrgh commented 3 years ago

I'm having the same issue trying to restore a 638mb snapshot to a new installation of hassos_rpi3-64-5.9.img.xz

When I try to upload a snapshot, I just get a spinning circle:

Then it says "connection lost", I refresh the page, and have to re-log in again. No error messages, nothing in the logs, no snapshots.

deftdawg commented 3 years ago

@blerrgh not sure if @ludeeus has plans to reopen/fix/test this issue anytime soon (especially being holidays season). Your best bet is to follow the workaround steps I posted up a bit, that'll at least get you back up and running. Tag me here if those steps give you grief. Good Luck!

blerrgh commented 3 years ago

@blerrgh not sure if @ludeeus has plans to reopen/fix/test this issue anytime soon (especially being holidays season). Your best bet is to follow the workaround steps I posted up a bit, that'll at least get you back up and running. Tag me here if those steps give you grief. Good Luck!

Thanks for the tip. I just installed the samba add-on, copied the snapshot over, and manually restored it. Took forever but restored successfully then.

home-assistant / frontend