restore to different namespace

[maty974]

Hi,

I have a condition where the source backup of an account (mymail@domain.com) has been made from an IMAP server with following namespaces

Name       Prefix     Delimiter
personal   "INBOX."   "."
other      (Not defined)
shared     (Not defined)

but the restore command for given mail account (mymail@domain.com) need to be made to another IMAP server with different namespaces:

Name       Prefix     Delimiter
personal   ""         "/"
other      (Not defined)
shared     (Not defined)

I currently planning to change mail hosting provider, but I keep the domain name, this is why I'm using restore command from the same mail account mymail@domain.com

Currently, if I use the restore command, I end up with all my folder structure outside of the INBOX and every subfolder separated from its parent folder :(

INBOX <~ destination imap folder
|- 'INBOX.folder1'
|- 'INBOX.folder1.sub1'

Is there any way with the restore command to specify/modify the destination namespaces ?

Thanks for your help 🙏

[joeyates]

Hi @maty974

I can see two ways of achieving this.

Rename the Folders After the Restore

Are you able to simply rename the folders that get created, after the restore?

Use migrate After Renaming the Source Account

1. Modify your configuration file manually (i.e. not via imap-backup setup) and rename your account to mymail-old@domain.com,
2. Change the name of the backup directory from mymail_domain.com to mymail-old_domain.com,
3. Set up a new account giving access to the new provider using imap-backup setup.
4. Run migrate:

imap-backup migrate mymail-old@domain.com mymail@domain.com --source-prefix=INBOX.

...let me know how it goes!

[maty974]

Hi @joeyates , thanks for your reply !

Unfortunately I can't rename the folders on the new imap server after restore, there's too much mailbox, I'm currently testing on one account but I have ~30 to migrate with a lot of subfolders hierarchy on the source imap accounts :/

For the second method, it seems to partialy works, only for the first level folder, the next subfolder hierarchy keep restoring each subfolders separated from it's parent folder

Is there a way to also specify destination delimiter, as in my case I moving from . delimiter to / ,

here's a test imap folders from source imap account

and here's after the migrate command on the destination imap server with / delimiter

Thanks for your help :pray:

[joeyates]

It makes sense to be able to specify delimiters (source and destination) both for migrate and for mirror.

I've started a branch to sort this out. Watch this space :)

[maty974]

Hi @joeyates !

Awesome ! thanks, I’ll wait for this new feature ;)

[joeyates]

@maty974 I've released version 9.0.0.rc1 with this update.

You can now supply migrate with the parameters --source-delimiter and --destination-delimiter.

Please give it a try and let me know how it goes.

[maty974]

@joeyates

It seems to works ! :smiley:

Even if I get a lot of WARN log message, not sure if this is normal ?

shan@imbkp:~$ /home/shan/imap-backup-9.0.0.rc1/bin/imap-backup migrate dummy@thedomain.com dummy@thedomain.fr -c /home/shan/config-gandi.json -r --source-delimiter='.' --destination-delimiter='/'

W, [2022-12-22T15:16:25.299443 #221233]  WARN -- : Folder 'INBOX/TEST_SUBS' does not exist on server
W, [2022-12-22T15:16:27.149209 #221233]  WARN -- : [INBOX/TEST_SUBS] uid: 1 (1/1) - append error: undefined method `uidvalidity' for "1671707425 1":String
W, [2022-12-22T15:16:27.541369 #221233]  WARN -- : Folder 'INBOX/Trash' does not exist on server
W, [2022-12-22T15:16:29.211912 #221233]  WARN -- : [INBOX/Trash] uid: 18 (1/2) - append error: undefined method `uidvalidity' for "1671707426 1":String
W, [2022-12-22T15:16:29.770543 #221233]  WARN -- : [INBOX/Trash] uid: 35 (2/2) - append error: undefined method `uidvalidity' for "1671707426 2":String
W, [2022-12-22T15:16:30.140518 #221233]  WARN -- : Folder 'INBOX/TEST_SUBS/SUB22' does not exist on server
W, [2022-12-22T15:16:31.814452 #221233]  WARN -- : [INBOX/TEST_SUBS/SUB22] uid: 1 (1/2) - append error: undefined method `uidvalidity' for "1671707427 1":String
W, [2022-12-22T15:16:33.132728 #221233]  WARN -- : [INBOX/TEST_SUBS/SUB22] uid: 2 (2/2) - append error: undefined method `uidvalidity' for "1671707427 2":String
W, [2022-12-22T15:16:34.437477 #221233]  WARN -- : [INBOX] uid: 15 (1/2) - append error: undefined method `uidvalidity' for "1671707421 1":String
W, [2022-12-22T15:16:34.996647 #221233]  WARN -- : [INBOX] uid: 16 (2/2) - append error: undefined method `uidvalidity' for "1671707421 2":String
W, [2022-12-22T15:16:35.366516 #221233]  WARN -- : Folder 'INBOX/Drafts' does not exist on server
W, [2022-12-22T15:16:37.223980 #221233]  WARN -- : [INBOX/Drafts] uid: 14 (1/1) - append error: undefined method `uidvalidity' for "1671707428 1":String
W, [2022-12-22T15:16:37.593223 #221233]  WARN -- : Folder 'INBOX/TEST_SUBS/SUB_MOVS' does not exist on server
W, [2022-12-22T15:16:39.684636 #221233]  WARN -- : [INBOX/TEST_SUBS/SUB_MOVS] uid: 4 (1/2) - append error: undefined method `uidvalidity' for "1671707429 1":String
W, [2022-12-22T15:16:40.429450 #221233]  WARN -- : [INBOX/TEST_SUBS/SUB_MOVS] uid: 5 (2/2) - append error: undefined method `uidvalidity' for "1671707429 2":String
W, [2022-12-22T15:16:40.799255 #221233]  WARN -- : Folder 'INBOX/Spam' does not exist on server
W, [2022-12-22T15:16:42.654489 #221233]  WARN -- : [INBOX/Spam] uid: 6 (1/1) - append error: undefined method `uidvalidity' for "1671707430 1":String
W, [2022-12-22T15:16:43.023389 #221233]  WARN -- : Folder 'INBOX/Sent' does not exist on server
W, [2022-12-22T15:16:44.877101 #221233]  WARN -- : [INBOX/Sent] uid: 6 (1/1) - append error: undefined method `uidvalidity' for "1671707431 1":String

The first time I've tried with the --source-prefix=INBOX. and the delimiter args, but it raise an error, after removed the source-prefix argument it was working

shan@imbkp:~$ /home/shan/imap-backup-9.0.0.rc1/bin/imap-backup migrate dummy@thedomain.com dummy@thedomain.fr -c /home/shan/config-gandi.json -r --source-prefix=INBOX. --source-delimiter='.' --destination-delimiter='/'

W, [2022-12-22T15:15:45.982589 #221227]  WARN -- : Folder 'TEST_SUBS' does not exist on server
W, [2022-12-22T15:15:47.993578 #221227]  WARN -- : [TEST_SUBS] uid: 1 (1/1) - append error: undefined method `uidvalidity' for "1671707422 1":String
W, [2022-12-22T15:15:48.363281 #221227]  WARN -- : Folder 'Trash' does not exist on server
W, [2022-12-22T15:15:50.069517 #221227]  WARN -- : [Trash] uid: 18 (1/2) - append error: undefined method `uidvalidity' for "1671707423 1":String
W, [2022-12-22T15:15:50.628598 #221227]  WARN -- : [Trash] uid: 35 (2/2) - append error: undefined method `uidvalidity' for "1671707423 2":String
W, [2022-12-22T15:15:50.999241 #221227]  WARN -- : Folder 'TEST_SUBS/SUB22' does not exist on server
W, [2022-12-22T15:15:52.670892 #221227]  WARN -- : [TEST_SUBS/SUB22] uid: 1 (1/2) - append error: undefined method `uidvalidity' for "1671707424 1":String
W, [2022-12-22T15:15:53.985903 #221227]  WARN -- : [TEST_SUBS/SUB22] uid: 2 (2/2) - append error: undefined method `uidvalidity' for "1671707424 2":String
W, [2022-12-22T15:15:54.357248 #221227]  WARN -- : Folder '' does not exist on server
Traceback (most recent call last):
        22: from /home/shan/imap-backup-9.0.0.rc1/bin/imap-backup:11:in `<main>'
        21: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/logger.rb:34:in `sanitize_stderr'
        20: from /home/shan/imap-backup-9.0.0.rc1/bin/imap-backup:12:in `block in <main>'
        19: from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
        18: from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        17: from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        16: from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        15: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli.rb:122:in `migrate'
        14: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli/migrate.rb:42:in `run'
        13: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli/folder_enumerator.rb:28:in `each'
        12: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli/folder_enumerator.rb:28:in `glob'
        11: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli/folder_enumerator.rb:28:in `glob'
        10: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli/folder_enumerator.rb:32:in `block in each'
         9: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/cli/migrate.rb:43:in `block in run'
         8: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/migrator.rb:15:in `run'
         7: from /home/shan/imap-backup-9.0.0.rc1/lib/imap/backup/account/folder.rb:45:in `create'
         6: from /usr/lib/ruby/2.7.0/forwardable.rb:235:in `create'
         5: from /usr/lib/ruby/2.7.0/net/imap.rb:481:in `create'
         4: from /usr/lib/ruby/2.7.0/net/imap.rb:1258:in `send_command'
         3: from /usr/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
         2: from /usr/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
         1: from /usr/lib/ruby/2.7.0/net/imap.rb:1276:in `block in send_command'
/usr/lib/ruby/2.7.0/net/imap.rb:1222:in `get_tagged_response':  Invalid mailbox name: Name is empty (0.001 + 0.185 + 0.184 secs). (Net::IMAP::NoResponseError)

I'll give it a try on a big mail account to see how it goes.

THanks again :pray:

[joeyates]

@maty974 I think there are a couple of things that are going wrong here.

Firstly, the "warning" append error: undefined method 'uidvalidity' for "1671707422 1":String is odd. It means that imap-backup is not picking up the net-imap gem (version >= 0.3.2), as required by the current version.

In fact, the last line of that output has /usr/lib/ruby/2.7.0/net/imap.rb as the path for 'net-imap', meaning it's using the version of 'net-imap' that comes with Ruby 2.7 - the 'standard library' version.

Could you try installing from source and see if things run more smoothly?

That problem seems to be obscuring another one related to the folder names that are being created, but I think it's best to clear up the 'net-imap' issue first.

[maty974]

@joeyates

I've followed the installing from source, but it seems to still load the net-imap from the standard library of Ruby.

To give it a try I've renamed the imap.rb of the ruby install path mv /usr/lib/ruby/2.7.0/net/imap.rb /usr/lib/ruby/2.7.0/net/imap.BKP_ ( I know this is bad :smiling_face_with_tear: ) please let me know if there is another "clean" way of forcing Gem to not load the std lib :P

A clean/full migrate command is running now, I don't see any WARN append error: undefined method 'uidvalidity' for ...

I'll let you know when it's done ;)

Thanks!

[maty974]

@joeyates

The migrate command is done, on a ~1.7Go account with a lot of folder/subfolders, I don't get any error/warning :smile: :+1: :tada:

The only thing that is quite annoying, and certainly not related to your tool, but each newly migrated folders appears in the new accout as unsubscribed and there are not visible in the webmail or mail client until you subscribe to the folders, and checking this box subscribe button on ~50+ folders is a pain :P

Anyway, thanks for your help, you have made my xmas holidays less stressed :clap:

[joeyates]

I'm glad that's working now. I'm using a Docker image to reproduce the problem with loading 'net-imap'. Looks like it doesn't load under Ruby 2.7, so I'll need to get that sorted out before the next release!