search

007revad / Synology_app_mover

Easily move Synology packages from 1 volume to another volume
MIT License
368 stars 26 forks source link

Trouble with package data - Container Manager, ABB, Hyper Backup, Surveillance Station #10

Closed jordancrawfordnz closed 9 months ago

jordancrawfordnz commented 10 months ago

Today I migrated all of my packages from my old 2.5" SSD to my new NVMe SSD's on my DS920+ today (DSM 7.2.1-69057 Update 3).

First of all - I wanted to say thank you for this project and for Synology_enable_M2_volume - both of these tools have made this process a lot easier (I wouldn't have purchased an NVMe if it wasn't for these tools!)

I suspect the issues I've noticed are similar to other issues like #9.

I also don't know how much can be done about the issues I mention - I don't know a lot about the internals of Synology configuration and some of the issues are very specific to a single package. I figured it might be best to mention this stuff here as it's easier to find than on reddit or off on my own blog.

Packages moved with no issues

Packages with issues

For reference, my old volume was /volume2 and my new volume was /volume3

Container Manager

I noticed that I still had about 40GB on the old SSD which I didn't expect.

I discovered a lot of this space was a /volume2/@docker directory. While I had a lot of old containers to prune, the surprised here was that even though Container Manager had been moved to my new volume, the Docker images appeared to remain stored on volume2.

I assume that there's some other configuration file which points to the actual Docker directory - presumably it'd need to update all references to the Docker directory and move the @docker directory itself.

To resolve my issue I:

Active Backup for Business

I dug through the rest of my volume2 directories and found a /volume2/@ActiveBackup directory which contained some files which looked like DBs.

The UI still worked and seemed to be using lock files and thing that were in this directory on volume2, despite the app now being installed on volume3.

I couldn't find any other configuration which seemed relevant in the @appdata, @appconf, etc directories. I decided to try a fresh install like I'd done for Docker - I moved the package back to volume2 and uninstalled it (without removing data). When I re-installed it I expected a fresh setup experience, but I found it was somehow still pointing at /volume2/@ActiveBackup and had retained my information.

To fix it I:

(I considered restoring my ABB settings from Hyper Backup but unfortunately it seems like ABB doesn't support backing up through Hyper Backup)

Hyper Backup

I saw a /volume2/@img_bkp_cache directory which seemed important for Hyper Backup. As far as I can tell, these are index files for the cloud backups (presumably so you can enquire about cloud backups without Hyper Backup having to connect to the cloud provider all the time). In my case, I backup configs to Google Drive. It seemed like this directory was still being updated by Hyper Backup even though I'd moved Hyper Backup to volume3.

I attempted a fresh install of Hyper Backup but I really didn't want to set everything up again. I think during the uninstall the @img_bkp_cache directory on volume2 got wiped.

When I re-installed Hyper Backup on volume2 and moved it across again to volume3 I had to re-index my config backup to Google Drive. As far as I can tell the @img_bkp_cache dir isn't all that important if you can't bring it across.

Surveillance Station

Similar to some of the other apps, there was a /volume2/@surveillance directory which seemed important for Surveillance Station. This directory was being used even though I'd moved the package to volume3. I couldn't find any settings for this.

For a fresh Surveillance Station install I:

I used Hyper Backup to restore my Surveillance Station config which got everything back to working.

I also noticed a /volume2/@ssbackup directory which also seemed to contain files relevant to Surveillance Station. Throughout the process above this directory disappeared - I don't think I deleted it and it didn't seem to matter, so hopefully it wasn't too important.

Let me know if you'd like any additional information on these issues. Again I'm not necessarily expecting this script to resolve all these, but I figured someone might know a way the script can automatically solve some of these problems or perhaps warn the user about troublesome packages.

007revad commented 10 months ago

Excellent detailed information. Thank you.

I do plan on checking packages for dependencies and moving them too, to deal with things like Synology Photos and NodeJS.

SnapshotReplication

I just noticed that SnapshotReplication has folders that aren't moved:

  1. @SnapshotReplication though this only contains an empty 'Plan folder (well it's empty for me).
  2. @sharesnap which appears to contain a lot of important data.

Just as well my script doesn't move because each volume that has snapshots enabled has their own data in their own @sharesnap folder.

Cloud Sync and USB Copy

It sounds like Cloud Sync and USB Copy are like Synology Drive. In the next version of syno_app_mover I've already added a message at the end that tells the user to open Synology Drive Admin Console and change the Synology Drive database to the new volume, then start Synology Drive from package center.

USB Copy has 2 files that could be what point to the volume the DB is stored on:

  1. /var/packages/USBCopy/etc/synopkg_conf/reg_volume which contains ["/volume1"].
  2. /var/packages/USBCopy/etc/setting.conf which contains repo_vol_path="/volume1"

Container Manager

Last week I did a lot of work trying to get Container Manager to move without issues. If I also move /volume#/@docker the containers are missing. And the images are listed but missing and need to be downloaded again.

I tried moving Container Manager. I tried cloning it using rsync. I tried uninstalling Container Manager and reinstalling it. I tried moving and copying the /volume#/@docker folder and updating the symlink that points to it. No matter what I tried I could relocate docker and have all the containers still working. I'll get back to this eventually.. but so far nobody has complained.

Active Backup for Business

It sounds like for Active Backup for Business I can do something that I tried with Container Manager:

  1. Backup /volume#/@ActiveBackup
  2. Uninstall Active Backup for Business.
  3. Reinstall Active Backup for Business on the target volume.
  4. Stop Active Backup for Business.
  5. Copy the backup of @ActiveBackup to the target volume.
  6. Start Active Backup for Business.

Hyper Backup

It sounds like I can probably do something similar for Hyper Backup:

  1. Backup /volume#/@img_bkp_cache
  2. Uninstall Hyper Backup.
  3. Reinstall Hyper Backup on the target volume.
  4. Stop Hyper Backup.
  5. Copy the backup of @img_bkp_cache to the target volume.
  6. Start Hyper Backup.

Surveillance Station

Again it sounds like I can probably do something similar for Surveillance Station:

  1. Backup /volume#/@surveillance
  2. Uninstall Surveillance Station.
  3. Reinstall Surveillance Station on the target volume.
  4. Stop Surveillance Station.
  5. Copy the backup of @surveillance to the target volume.
  6. Start Surveillance Station.

Though I just found 2 files that I could edit after moving Surveillance Station and /volume#/@surveillance

  1. /var/packages/SurveillanceStation/etc/synopkg_conf/reg_volume which contains ["/volume1"]
  2. /var/packages/SurveillanceStation/etc/settings.conf which contains active_volume="/volume1"

SynologyApplicationService

SynologyApplicationService has an sqlite database here: /volume1/@SynologyApplicationService/notification/notification_db_daemon.sqlite

For me this database only contains DownloadStation entries July, Aug and Dec 2023. And they all appear to only be DSM Notifications.

Though I should move /volume1/@SynologyApplicationService and edit: /var/packages/SynologyApplicationService/etc/settings.conf which contains volume="/volume1/@SynologyApplicationService

jordancrawfordnz commented 10 months ago

Hi David - glad to see you’ve already seen some of the problems I mentioned!

@SnapshotReplication though this only contains an empty 'Plan folder (well it's empty for me).

Ah yeah I also saw a few things on my unit which just had empty ‘plan’ directories - for me it was @SnapshotReplication and @Virtualization. I ignored them as it didn’t seem important.

@sharesnap which appears to contain a lot of important data.

Ah yeah - I think I left this alone too. I was only going for anything which looked important for packages.

I’m curious to see how you get on with Active Backup for Business, Hyper Backup and Surveillance Station.

I don’t think I actually tried just moving the @surveillance, @ActiveBackup etc directories over - I was finding that even when the package had moved to the new volume it was still using the files from the old volume. I suspect that there might be another config file somewhere which tells it which volume to point at - but maybe it’s simpler than I expected and would scan for those directories on its installed volume.

My uncertainty about this is the reason why I went with a completely fresh ABB instance (so that the install pointed at the @ActiveBackup directory on the new volume) and then moved in the data from the old volume.

007revad commented 9 months ago

@jordancrawfordnz

NodeJS - I had to hit 'repair' in Package Center but it seemed to sort it out fine.

When I do that it repairs itself back to the original volume. Interestingly even though NodeJS shows the Repair button it is actually running.

I tried uninstalling NodeJS and installing it on the target volume, but you can't uninstall a package that other packages depend on. Which could be why moving the package triggered the Repair status.

I'm working my way through each Synology package:

  1. Installing package.
  2. Changing settings.
  3. Create data.
  4. Move package.
  5. Check my changed settings and data still exist.

Here's what I've tested or got working so far: https://github.com/007revad/Synology_app_mover/blob/test/README.md

007revad commented 9 months ago

I've got the Surveillance Station moving okay, but in the process I think I may changed the owner and group of the @surveillance folder.

Can you tell me what these commands return:

ls -l /var/packages/SurveillanceStation/target/@surveillance
ls -l /volume3 | grep @surveillance

To get surveillance station working I had to climb into the roof to get an old camera I'd left up there years ago. I am surprised it still works as it is covered in spider webs, mould and dust and it appears to have gotten wet at some point.

For the last few days I've been installing apps, changing settings, moving the apps, then checking they start and my settings are still there.

Ignoring all the community packages there are around 100 packages available in package center. I've got about 36 to go https://github.com/007revad/Synology_app_mover/blob/test/README.md

007revad commented 9 months ago

I don’t think I actually tried just moving the @surveillance, @ActiveBackup etc directories over - I was finding that even when the package had moved to the new volume it was still using the files from the old volume. I suspect that there might be another config file somewhere which tells it which volume to point at - but maybe it’s simpler than I expected and would scan for those directories on its installed volume.

Some packages have /var/packages//etc/synopkg_conf/reg_volume which contains either ["/volume1"] or ["/volume1","/volume1"]

jordancrawfordnz commented 9 months ago

Hi @007revad - apologies for the late reply!

Node

My recollection and notes are a little rusty from last week but I just wanted to provide some clarity.

  1. Synology Photos was one of the first things I moved over with Synology_app_mover
  2. When I used Synology_app_mover to move Node, it shut down Synology Photos.
  3. I believe at this point Package Center said that Node needed to be repaired.
  4. I believe I attempted to start Synology Photos in with Node in this 'needs repair' state and it said it started but the UI couldn't load.
  5. When I repaired Node it seemed to re-install it onto the new Volume3 - I've had a reboot or two since removing the old Volume2 and it seems to be fine.

Surveillance Station

Can you tell me what these commands return:

Of course - 

jordancrawford@SpaceStation:~$ ls -l /var/packages/SurveillanceStation/target/@surveillance
lrwxrwxrwx 1 SurveillanceStation SurveillanceStation 22 Jan 13 14:06 /var/packages/SurveillanceStation/target/@surveillance -> /volume3/@surveillance
jordancrawford@SpaceStation:~$ ls -l /volume3 | grep @surveillance
drwxr-xr-x  1 SurveillanceStation SurveillanceStation 572 Jan 21 17:06 @surveillance

To get surveillance station working I had to climb into the roof to get an old camera I'd left up there years ago. I am surprised it still works as it is covered in spider webs, mould and dust and it appears to have gotten wet at some point.

Oh wow - that's dedication!

Config location

Some packages have /var/packages//etc/synopkg_conf/reg_volume which contains either ["/volume1"] or ["/volume1","/volume1"]

Ahh interesting!

I had a look at a few -

Active Backup for Business

It seems like ActiveBackup (/var/packages/ActiveBackup) uses some symlinks for this -

jordancrawford@SpaceStation:/var/packages/ActiveBackup$ ls -lah
total 132K
drwxr-xr-x  5 root root 4.0K Jan 13 17:40 .
drwxr-xr-x 34 root root 4.0K Jan 18 21:38 ..
drwxr-xr-x  3 root root 4.0K Jan 13 17:40 conf
-rw-r--r--  1 root root    0 Jan 13 17:40 enabled
lrwxrwxrwx  1 root root   30 Jan 13 15:24 etc -> /volume3/@appconf/ActiveBackup
lrwxrwxrwx  1 root root   30 Jan 13 15:24 home -> /volume3/@apphome/ActiveBackup
-rw-r--r--  1 root root 110K Jan 13 15:24 INFO
drwxr-xr-x  2 root root 4.0K Jan 13 15:23 scripts
lrwxrwxrwx  1 root root   31 Jan 13 15:24 share -> /volume3/@appshare/ActiveBackup
lrwxrwxrwx  1 root root   31 Jan 13 15:24 target -> /volume3/@appstore/ActiveBackup
lrwxrwxrwx  1 root root   30 Jan 13 15:24 tmp -> /volume3/@apptemp/ActiveBackup
lrwxrwxrwx  1 root root   30 Jan 13 15:24 var -> /volume3/@appdata/ActiveBackup

Surveillance Station

Surveillance Station also appears to use symlinks, but also has a etc/settings.conf.

jordancrawford@SpaceStation:/var/packages/SurveillanceStation$ ls -lah
total 100K
drwxr-xr-x  5 root root 4.0K Jan 13 17:40 .
drwxr-xr-x 34 root root 4.0K Jan 18 21:38 ..
drwxr-xr-x  2 root root 4.0K Jan 13 17:40 conf
-rw-r--r--  1 root root    0 Jan 13 17:40 enabled
lrwxrwxrwx  1 root root   37 Jan 13 14:06 etc -> /volume3/@appconf/SurveillanceStation
lrwxrwxrwx  1 root root   37 Jan 13 14:06 home -> /volume3/@apphome/SurveillanceStation
-rw-r--r--  1 root root  79K Jan 13 14:06 INFO
drwxr-xr-x  3 root root 4.0K Jan 13 14:05 scripts
lrwxrwxrwx  1 root root   38 Jan 13 14:06 share -> /volume3/@appshare/SurveillanceStation
lrwxrwxrwx  1 root root   38 Jan 13 14:06 target -> /volume3/@appstore/SurveillanceStation
lrwxrwxrwx  1 root root   37 Jan 13 14:06 tmp -> /volume3/@apptemp/SurveillanceStation
lrwxrwxrwx  1 root root   37 Jan 13 14:06 var -> /volume3/@appdata/SurveillanceStation
drwxr-xr-x  2 root root 4.0K Jan 13 14:05 WIZARD_UIFILES

Interestingly I note /var/packages/SurveillanceStation/etc/settings.conf has a active_volume="/volume1" setting which is interesting - I was concerned at first but then I remembered that this is the volume I store my recordings on, so it's probably fine.

007revad commented 9 months ago

I've released the new version: https://github.com/007revad/Synology_app_mover/releases/tag/v2.0.6