Restore menu option doesn't work on new server?

[thunderstorm654]

Hi, me again unfortunately!! I'm trying to migrate to a new Hetzner server as the spec was better than my old server at a cheaper price :). I have 3 days to complete migration before I have to pay for the old one again. I thought this would be easier than the Gooby > Omni upgrade but apparently not :(

I've got omni installed and omnimount is up and running and says 'healthy'. I tried to follow the migrating instructions here: https://www.omnistream.cloud/omni-basics/backup-and-restore#h.2cxhk4px5kko

But I'm stuck on the last bit, restore backup from the menu. When I go into omni menu > utilities > restore nothing happens. Well, there's a brief flash of the screen, like something tries to do something in the background, but nothing loads or displays, I just stay stuck on the menu - the below screen, which will just flash very quickly then stay there:

I can't find any log files to help with this, any ideas?

The only suspect thing I've noticed is if I do omni down / omni up, I see this nasty error:

Omni up:

I don't know if the jq error thing is causing trouble?

Also, I had automatic DNS management on my old server, will this get reconfigured if I eventually get to restoring?

Totally stuck now and not sure what to try next. Any advice welcome! Thanks :)

Edit: looks like the other open issue #39 has the same error coming up, wonder if they're related in some way...

[bifa2]

I've had a scan through the codebase for jq references and I suspect it's something in check-dns but I could be wrong. Bash isn't my strength, nevermind trying to debug it :)

[bifa2]

Actually, your screenshots are saying DNS management is disabled. No action performed. so it should be exiting before jq even runs. Hmmm.

[bifa2]

I wonder if when check-dns, del-dns run, the third line in those files (source ${OMNIHOME}/bin/dns_functions) both imports the functions in that file but also runs lines 81-84 at the bottom?

https://github.com/kelinger/OmniStream/blob/main/bin/dns_functions#L81

${CF_EMAIL} and ${CF_KEY} won't be set if you're not using CloudFlare management? No email/key will result in no return from the API and thus jq won't be able to iterate over anything?

Perhaps try commenting those lines out @thunderstorm654 and trying again, but equally wait for someone who knows what they're doing 😄

[thunderstorm654]

Oops posted with wrong ID :)

Hey @bifa2 I populated ${CF_EMAIL} and ${CF_KEY} and that made the jq error go away. So there's a small bug there hopefully the guys here can fix :) Thanks for working this out!

However the restore menu still won't work! Still just flashes quickly and doesn't load. It opens up fine on my old server. I can't spot anything obvious that I'm missing by looking at the restore script https://github.com/kelinger/OmniStream/blob/main/bin/restore, but I don't really know this language that well :). I checked that my mnt path can see the Backups folder and its showing up there. So don't think it's an issue with my mount. Stumped!

[kelinger]

@thunderstorm654

On your new server, if you have the basic OmniStream up and do a

omni mounts
cd cloud
cd Backups
ls -l

Do you see your old server and the backups from it?

[thunderstorm654]

@thunderstorm654

On your new server, if you have the basic OmniStream up and do a

omni mounts
cd cloud
cd Backups
ls -l

Do you see your old server and the backups from it?

@kelinger yes both the old and new server are listed, the mount itself seems to be working great.

[thunderstorm654]

Here's a short video clip of what happens (not much!!) When trying to choose "restore" (I'm pressing enter a few times!)

https://user-images.githubusercontent.com/16837607/211187350-e475e14e-af56-44c9-ad23-6d0305ef3188.mp4

[thunderstorm654]

Update: the restore menu now doesn't work on my old server either - exactly the same behaviour. That's really odd as it was working on Friday. Only change that's happened is the backup occurred from the new server. Not touched old server config at all. Makes no difference to old or new if I move the new server backup folder out (so only one server backup present) either.

Been staring at the code for the restore and just don't get it - the Backups directory is there, nothing appears wrong - but the script just won't run at all it seems. All the other menu options I've tried are functional but this one.

[kelinger]

Well, this is definitely a bug but I have to wonder if I'm maybe not quoting a variable or something like that. I haven't posted any code changes recently--especially to this part of the process--and the fact that it worked until Friday on your old server is perplexing. Do you perhaps have a space or "unusual" character in your server name(s)?

Can you respond with a list of the directories under the ".../cloud/Backup" on your cloud mount?

[thunderstorm654]

Well, this is definitely a bug but I have to wonder if I'm maybe not quoting a variable or something like that. I haven't posted any code changes recently--especially to this part of the process--and the fact that it worked until Friday on your old server is perplexing. Do you perhaps have a space or "unusual" character in your server name(s)?

Can you respond with a list of the directories under the ".../cloud/Backup" on your cloud mount?

It is indeed perplexing haha love that! The only "special"-ish character is a - but that was in the old server name.

The directories (pasting from ls -l) are:

thunderstorm@hetzstorm:~/OmniStream/mnt/cloud/Backups$ ls -l
total 4
drwxr-xr-x 3 thunderstorm thunderstorm 4096 Jan  8 04:15 hetzstorm
drwxrwxrwx 1 thunderstorm thunderstorm    0 Oct 11 22:45 ss-ex42deb

Edit: I did double check that removing the hyphen doesn't fix it, unfortunately it doesn't :(

[thunderstorm654]

Is there a way to run the restore another way perhaps? I did try calling the restore script directly via it's path in bin, and that also results in nothing happening too (expected, but consistent at least ha). If not what do you think the easiest way to move my stuff over would be? I wish I was better at bash scripting, unfortunately I work with Microsoft stuff and PowerShell or C# and while I get the general idea of what it's doing I'm not good at the syntax etc required here!

[kelinger]

Well, the "-" certainly shouldn't be causing any issues. But, just to test, I created a directory with a dash in it (actually used the same name you were using) and I got further than your video shows so it's still a head scratcher.

Yes, there is actually an easy way to restore manually (or semi-manually). I want to wait for just a few more back-and-forth questions, though, lest we may never find the actual issue. But rest assured, we'll get you back up and running one way or another!

In the meantime... let me check a few more things.

[thunderstorm654]

Well, the "-" certainly shouldn't be causing any issues. But, just to test, I created a directory with a dash in it (actually used the same name you were using) and I got further than your video shows so it's still a head scratcher.

Yes, there is actually an easy way to restore manually (or semi-manually). I want to wait for just a few more back-and-forth questions, though, lest we may never find the actual issue. But rest assured, we'll get you back up and running one way or another!

In the meantime... let me check a few more things.

No worries! Thanks for investigating. It's definitely really odd. As far as I know, it's the only menu choice that's playing tricks. Happy to try out anything you need to assist :)

[kelinger]

I don't know if this is good or bad news. I created a new instance with the basic Debian installation. Then, I ran the OmniStream setup script. After it installed all the Debian components, it installed Docker and Compose. When it got to the point where it was presenting dialogs, it walked me through the email address, domain name, and rclone setup. Then, when it asked if I wanted to do a restore, I said yes, and it mounted my cloud volume and listed the three servers I have backups from.

I believe that even this is further than you got, though. When it was doing the installation or setting up rclone, did you see any errors flash by on the screen?

[thunderstorm654]

I don't know if this is good or bad news. I created a new instance with the basic Debian installation. Then, I ran the OmniStream setup script. After it installed all the Debian components, it installed Docker and Compose. When it got to the point where it was presenting dialogs, it walked me through the email address, domain name, and rclone setup. Then, when it asked if I wanted to do a restore, I said yes, and it mounted my cloud volume and listed the three servers I have backups from.

I believe that even this is further than you got, though. When it was doing the installation or setting up rclone, did you see any errors flash by on the screen?

Ugh, bad news from here. I reset the server and went again from scratch. All I did before running OmniStream setup was install a Debian image:

echo x | installimage -p /boot:ext3:1G,/:ext4:all -l 0 -r yes -i images/Debian-1011-buster-64-minimal.tar.gz -a -n hetzstorm && reboot

Then did: apt update apt install sudo

Then ran OmniStream. I had fun with the rclone config again, when I paste in the conf contents it doesn't then prompt me to select which mount and errors trying to read the config folder (permissions). I had that months back when I did Gooby > OmniStream too. So I had to grant permissions to the config folder:

sudo chown -R $USER:$USER ~/.config sudo chown -R $USER:$USER $OMNIHOME/configs

and then Omni edit and set the rclone mount name/service, as these are missing. Then Omni up brings up the mount successfully.

Went immediately to try and restore after the mount is working and same behaviour... Brief screen flash.

:( Damn! Can't get much more minimal than this but there must be something weird about hetzner maybe?! Or the Debian image installed?

[thunderstorm654]

I've tried using the latest Debian image available to install on the Hetzner server (Debian-1106-bullseye-amd64-base.tar.gz) and the only difference with that one is sudo is already installed and working.

However, with the OmniStream install I still face the rclone config file not popping up to select my mount after pasting/saving (I wonder if this works in a similar way to the restore menu?) , so have to manually update config folder perms as per last reply to get the mount up.

Then frustratingly the restore menu still doesn't work, exactly the same behaviour :( bangs head on desk is there another way to try it? Since this option is clearly not going to work for me for some weird reason we may find hard to answer!

[kelinger]

Are you running any of the OmniStream functions with sudo? You should not be and may explain why the runtime doesn't have access to the rclone.conf file.

One other thing to try is to paste your rclone.conf manually into ~/.config/rclone.conf before running the setup (again, don't use sudo). Then, when setup asks if you want to edit the file, say yes, let it open, and just save/exit.

[thunderstorm654]

Are you running any of the OmniStream functions with sudo? You should not be and may explain why the runtime doesn't have access to the rclone.conf file.

One other thing to try is to paste your rclone.conf manually into ~/.config/rclone.conf before running the setup (again, don't use sudo). Then, when setup asks if you want to edit the file, say yes, let it open, and just save/exit.

No, I follow the install instructions and create my user and switch to that to install. Not running anything OmniStream with sudo at all :). For some reason, my user account doesn't have access to the .config folder, so the OmniStream setup can't start the mount as it cannot access the conf file. So I only use sudo to grant access to the folder.

Any ideas RE the restore? Thanks :)

[thunderstorm654]

Sorry, this is a long response, but wanted to document where I got to! TLDR, I got a restore to work by copying and modifying the script to avoid one specific dialog, then there's a question for you about backups at the bottom :)

---

So, I've decided to take it apart line by line (I took a copy of the restore script into the root of my home directory, not messing with the one in bin :)) and work out where the restore script is falling over. BTW, it started working again on my old server (which makes me wonder was I going mad the other day and somehow looking at the wrong thing, as nothing changed!).

It falls over on my new server on this line, (https://github.com/kelinger/OmniStream/blob/main/bin/restore#L48):

CHOICE=$(dialog --clear \
    --backtitle "Restore OmniStream" \
    --title "Server selection" \
    --menu "Restore from which servers?" \
    0 0 0 \
    "${SERVERS[@]}" \
    2>&1 >$TERMINAL)

This line fails to create the dialog, so if the script proceeds any further it just stops/fails - no visible error but it just does nothing (as per the 'flash' when called from the menu route).

I checked the values set in variables up to this point (echo'ing them out) and both old & new servers.. both have the correct info in SERVERS and TERMINAL. I wondered if dialog was being funky, as I can't see anything wrong with this line at all, and it works on my old server, all your servers.. WTF! I checked the version I had installed of the dialog package, my old server had a slightly older version, so in some desperation I downgraded the one on my new one... no difference! LOL.

In the end, I have managed to get the restore working by commenting out this line and just manually setting SERVERS variable to my old server name. Then if I run my modified copied script manually, it proceeds and all the other menu options come up and work just fine (select which apps to restore, select diffs, all good!). I am really confused as to why that particular line doesn't work on my new server... it's hardly rocket science is it :). With the same dialog package version installed and same input it literally makes no sense. I'm feeling inclined to assume it's got to be something bizarre with the dialog package itself, which we'll probably never understand and it's seemingly isolated to just me and one server - despite about 5 full resets & OS reinstalls..!! The only difference between the servers is old one was set up with Gooby and upgraded to OmniStream, vs new is a fresh OmniStream install & OmniStream restore.

Unless, you have any other ideas why dialog would be bugging out here - without being able to replicate it, I'm all too familiar with how annoying that can be to understand :)

I'm happy to just utilise my slightly modified script for now, I think unless anyone else has an issue in future it's probably something bizarre only I'm getting.

I only have one question hopefully you can answer, RE the differential backups. I have forced another backup from my old server today and it has been uploaded successfully into gdrive. However when that's restored, the progress in Plex/Emby (which both have the differential backups enabled) is a couple of days out (I think Sunday). I tried doing another backup again, to double check the diff is getting updated (and it appears to), but same result of couple days out on progress. Am I correct in thinking I can force the backup to be totally re-done if I remove the snapshots from the 'snapshots' folder? It's not a big deal, I just have a few users who might (mildy) complain ;)

[kelinger]

Well, that's a good find and it would mean (I suppose) that whatever was in the SERVERS variable that you avoided by hard coding it was causing the issue. Is there any chance you could add a "echo $SERVERS ; exit" to the script to see what values it was (trying) to use?

Differential:

Many of the containers use databases like MySQL and file-level backups on databases aren't really the best way to do things. In most cases, it won't cause issues because these aren't exactly apps that are pounding sequential updates from thousands of sources every second (think: Amazon's shopping cart). The "best" way to backup is to shutdown the container and then run backup (which, granted, isn't always possible nor desirable). The second best is to use a program that actually uses the SQL engine's backup process to extract the data to a file(s) which are then backed up. This isn't that kind of project and would add a ton of complexity.

So, instead, my personal practice is to use the cron job backup that OmniStream installs and that works most of the time. But, if I'm going to wipe my server clean or move to a new server, I'll shut down all containers except OmniMount and then run backup manually.

Additionally, I believe that Plex, Emby, and most of the *arrs also do their own backups (see related documentation for their containers) so that even if your backup is bad due to a database corruption, there's another database file stored somewhere in their configs that can be used to get back to (or close to) the most recent image.

And actually, none of this has much to do with differentials at all. It's just that with differentials, there are twice as many opportunities for a file like a database to be "active" while being backed up: once with the base image backup and again with the daily differential.

I suppose if your viewing habits are predictable, you could create a backup script in the plugins (backup-start and backup-end are supported) that would do the shutdown and startup of the containers so that each backup is clean. However, in my personal case, this works great until you have an all night movie-thon and the backup kicks off at the worst possible time.

[kelinger]

@thunderstorm654 - have I answered all your questions (related to this topic)? If so, can we close it out?

[thunderstorm654]

Hey @kelinger apologies I never got back to you with your previous reply! Sorted it all out in the end.

The servers variable was exactly the same on both, it really was a strange mystery but hopefully it won't show up again now I'm up and running 😄

Thanks again for all your replies and awesome work on OmniStream. Really happy with it, it's really easy to use and keep containers updated 😎