don't assume the sep character is always required

tremby commented 4 years ago

General informations

system/distribution (with version): Ubuntu 18.04.4 LTS
offlineimap version (offlineimap -V): offlineimap v7.1.5, imaplib2 v2.57 (system), Python v2.7.17, OpenSSL 1.1.1 11 Sep 2018
Python version: 2.7.17
server name or domain: -
CLI options: -

Configuration file offlineimaprc

REMOVE PRIVATE DATA.
[general]
accounts = test

[Account test]
localrepository = Local
remoterepository = Mpro

[Repository Local]
type = Maildir
localfolders = ~/mail-test

[Repository Mpro]
type = IMAP
remotehost = 192.168.1.6
ssl = no
remoteuser = test
remotepass = test
readonly = True
maxconnections = 3
folderfilter = lambda dirname: dirname not in [
    'wastebasket',
    ]
maxconnections = 3

pythonfile (if any)

none

Logs, error

REMOVE PRIVATE DATA.
$ offlineimap -c mprofetch.conf --info
OfflineIMAP 7.1.5
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.57 (system), Python v2.7.17, OpenSSL 1.1.1  11 Sep 2018
  imaplib2: 2.57 (system)
Remote repository 'Mpro': type 'IMAP'
Host: 192.168.1.6 Port: None SSL: False
Establishing connection to 192.168.1.6:143 (Mpro)
Server welcome string: * OK 6.abbeypress.net MsgServe v8.01S IMAP server ready.
Server capabilities: ('IMAP4REV1', 'UIDPLUS')

folderfilter= lambda dirname: dirname not in [
'wastebasket',
]

Folderlist:
 freefind
 INBOX
 outgoing.mail
...

$ offlineimap -c mprofetch.conf
OfflineIMAP 7.1.5
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.57 (system), Python v2.7.17, OpenSSL 1.1.1  11 Sep 2018
Account sync test:
 *** Processing account test
 Establishing connection to 192.168.1.6:143 (Mpro)
 Creating folder .o.u.t.g.o.i.n.g...m.a.i.l.[Local]
 Creating new Local Status db for Local:.o.u.t.g.o.i.n.g...m.a.i.l.
 Creating folder .f.r.e.e.f.i.n.d.[Local]
 Creating folder .I.N.B.O.X.[Local]
 Creating new Local Status db for Local:.I.N.B.O.X.
...

Steps to reproduce the error

run offlineimap --info, see that the remote folderlist has things like INBOX, outgoing.mail, freefind
run offlineimap, see that they are downloaded to folders like .I.N.B.O.X., .o.u.t.g.o.i.n.g...m.a.i.l., .f.r.e.e.f.i.n.d.

Why are these folders being saved like this?

If I set sep to for example @ I then get things like @I@N@B@O@X@.

I don't know if it's related, but I am also noticing that outgoing.mail is being skipped entirely, because its name contains the separator. If I change sep to @ its contents then get downloaded. There is no mailbox on the remote called just outgoing, in case that's relevant.

The remote server is MsgServe running on RISC OS. I wonder if the bug lies on their end, but when I probe the server manually by speaking IMAP to it over telnet, I don't see anything unexpected.

nicolas33 commented 4 years ago

That's weird. The issue is likely not the server but offlineimap. Could you try the latest stable v7.3.3 please?

tremby commented 4 years ago

Same thing.

$ .local/bin/offlineimap -c mprofetch.conf --info                                     ↵0·2:20:52
OfflineIMAP 7.3.3
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.57 (system), Python v2.7.17, OpenSSL 1.1.1  11 Sep 2018
  imaplib2: 2.57 (system)
Remote repository 'Mpro': type 'IMAP'
Host: 192.168.1.6 Port: None SSL: False
Establishing connection to 192.168.1.6:143 (Mpro)
Server welcome string: * OK 6.abbeypress.net MsgServe v8.01S IMAP server ready.
Server capabilities: ('IMAP4REV1', 'UIDPLUS')

folderfilter= lambda dirname: dirname not in [
'wastebasket',
]

Folderlist:
 freefind
 INBOX
 outgoing.mail
...

$ .local/bin/offlineimap -c mprofetch.conf
OfflineIMAP 7.3.3
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.57 (system), Python v2.7.17, OpenSSL 1.1.1  11 Sep 2018
Account sync test:
 *** Processing account test
 Establishing connection to 192.168.1.6:143 (Mpro)
 Creating folder .o.u.t.g.o.i.n.g...m.a.i.l.[Local]
 Creating new Local Status db for Local:.o.u.t.g.o.i.n.g...m.a.i.l.
 Creating folder .f.r.e.e.f.i.n.d.[Local]
 Creating new Local Status db for Local:.f.r.e.e.f.i.n.d.
 Creating folder .I.N.B.O.X.[Local]
 Creating new Local Status db for Local:.I.N.B.O.X.

Version info is now

offlineimap v7.3.3, imaplib2 v2.57 (system), Python v2.7.17, OpenSSL 1.1.1  11 Sep 2018

nicolas33 commented 4 years ago

I wonder the IMAP server is not correctly providing the sep char (returning empty string). What's the IMAP server/proxy/whatever at 192.168.1.6?

tremby commented 4 years ago

It's MsgServe running on RISC OS. Is there something I can send it over telnet to find out if that's the issue?

nicolas33 commented 4 years ago

Sorry, this info was in your intial post. Do you have a subfolder 'mail' in 'outgoing' (outgoing/mail)? If so, this issue is likely a str type of a python variable we expect as a list/tuple. My guess is that we do something like a join() on the wrong type.

Is there something I can send it over telnet to find out if that's the issue?

The sep char is provided by the server in the response of a LIST command.

tremby commented 4 years ago

No, there's no such parent directory. The separator is being given as NIL.

[bjn@elastic:~]$ telnet 192.168.1.6 143                                                             ↵0·15:55:47
Trying 192.168.1.6...
Connected to 192.168.1.6.
Escape character is '^]'.
* OK 6.abbeypress.net MsgServe v8.01S IMAP server ready.
a login brian a
a OK login command successful.
. list "" "*"
* LIST (\NoInferiors) NIL "INBOX"
* LIST (\NoInferiors) NIL "deferred.mail"
* LIST (\NoInferiors) NIL "freefind"
* LIST (\NoInferiors) NIL "inMarcel(1999-2007)"
* LIST (\NoInferiors) NIL "list-admin"
* LIST (\NoInferiors) NIL "outgoing.mail"
* LIST (\NoInferiors) NIL "sentMarcel(1999-2007)"
* LIST (\NoInferiors) NIL "spam"
* LIST (\NoInferiors) NIL "tests"
* LIST (\NoInferiors) NIL "Trash"
* LIST (\NoInferiors) NIL "weekly"
* LIST (\NoInferiors) NIL "wastebasket"
. OK list command successful.

I believe the separator should be . since other users on this system have mailboxes like people.alice, people.bob, which appear hierarchically under people in the GUI client on this system.

But I don't much mind for my archival/migration purposes if it all comes back as a flat list, some names just having "." as part of their name, and according to the RFC you linked this is the way to do that.

nicolas33 commented 4 years ago

We don't correctly support the "no hierarchy" LIST response with NIL as provided above. https://tools.ietf.org/html/rfc3501#section-7.2.2

tremby commented 4 years ago

I've posted full details of the process I used to migrate mail from a legacy IMAP server to Gmail with OfflineIMAP, including the various workarounds involved, here: https://gist.github.com/tremby/a4caaa86bb69e6af052a55b77eae48be -- just in case it helps anyone in future!

nicolas33 commented 4 years ago

@tremby What about writing a blog post at offlineimap.org (see https://github.com/OfflineIMAP/offlineimap.github.io/tree/master/_posts).

OfflineIMAP / offlineimap