Add support for mailservers using a dot as IMAP hierarchy delimiter

[mrzool]

Hello and thanks for this tool.

I keep getting this error:

NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.)

I'm trying this with an empty mailbox that I set up for testing purposes. I'm positive that INBOX is the correct IMAP prefix to use. --list_boxes gives me this result:

"INBOX"                                 
  +- "Sent Messages""                   #UnMarked
  +- "Drafts"                           #UnMarked   #Drafts

Any idea on what the problem might be?

[rgladwell]

Which mail provider is this for?

[mrzool]

Thanks for answering! I tested further yesterday and I can be more specific now.

Which mail provider is this for?

I'm testing this on a Hetzner mailserver with these capabilities:

openssl s_client -connect mail.your-server.de:993 -crlf
[...]
OK [CAPABILITY IMAP4 IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION]

The problem arises only when using the -r flag and scanning a directory structure (exported from Mail.app) for mbox files. The upload works fine with a single mbox file is provided, together with a specified box path.

I'm no IMAP expert and I might be wrong, but I'm under the impression that the notation used by your tool to specify the box path on the server might be causing the issue with this particular mailserver.

If I provide a single mbox and specify a box path in the destination using a . separator, the upload works fine and non-existing directories get created as needed.

imap_upload.py emails/directory/subdirectory.mbox/mbox imaps://testing@example.de:password@mail.your-server.de:993/INBOX.Newdir
Connecting to mail.your-server.de:993.
Uploading to INBOX.Newdir...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)
  2/41  24.7 kB  Second Email    OK (1 sec)
[...]

Note the line Uploading to INBOX.Newdir..., where the path that I specified with a dot separator is used.

But if I use the -r flag to scan a path, imap-upload passes the filesystem path with a standard Unix / separator to the server. This seem to cause the namespace error:

imap_upload.py -r emails imaps://testing@example.de:password@mail.your-server.de:993
Connecting to mail.your-server.de:993.
Found mailbox at emails/directory/subdirectory.mbox/mbox...
Uploading to directory/subdirectory...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)
  2/41  24.7 kB  Second Email    OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)

Again I might be wrong, but this is the behavior I have been observing. The configuration of my IMAP server might have something to do with it? The IMAP options altnamespace and unixhierarchysep came up in my research.

Happy to test further. Again, thanks for your help!

Update: Another test I just did seems to confirm this. If I append a box path with a / separator to the destination the upload fails in the same way.

imap_upload.py emails/directory/subdirectory.mbox/mbox imaps://testing@example.de:password@mail.your-server.de:993/INBOX/Newdir
Connecting to mail.your-server.de:993.
Uploading to INBOX/Newdir...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)
  2/41  24.7 kB  Second Email    OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)
[...]

[rgladwell]

Hey @mrzool thanks for the detailed feedback.

The trick with imap-upload is that it often requires support for different mail servers as some have differing conventions.

It sounds as though we need to add a Hetzner CLI argument to enable alternatives to the / separator. Or perhaps an argument that changes the file separator character.

Do you know which character the separator should be?

[mrzool]

From what I've read it seems like there are only two hierarchy delimiters in use today — / if unixhierarchysep is on and . if it's off. Different IMAP servers adopt different defaults.

The absolute ideal solution would be to detect the delimiter in use on a particular server automatically. imaplib seems to provide a way to find this out. Not sure how complicated this would be to implement in your script.

The next best option would be to assume a / delimiter — that seems to be the predominant anyway, as it allows dots in usernames (john.doe@example.org) without having to resort to internal IMAP hacks on the server (see the first warning here) — and just provide a flag to override it with .. This way a user would probably be able to get imap-upload to work through trial and error. Still a win!

It sounds as though we need to add a Hetzner CLI argument to enable alternatives to the / separator.

This is in no way specific to Hetzner, the dot separator is apparently still in wide use. The benefits of implementing a way to deal with this would go beyond this particular use case.

Whatever you choose to do, I'm very grateful that you're looking into it. It's a huge help.

[rgladwell]

If we added a --separator-char so you could execute something like this: separator

$ python imap_upload.py --separator-char '.'

Could this fix the issue for you?

[mrzool]

Could this fix the issue for you?

I think so! I would test it right away.

[rgladwell]

Close for PR: https://github.com/rgladwell/imap-upload/pull/18