foxcpp / maddy

✉️ Composable all-in-one mail server.
https://maddy.email
GNU General Public License v3.0
5.07k stars 244 forks source link

Maildir storage backend #6

Open emersion opened 6 years ago

emersion commented 6 years ago

Probably maildir

xeoncross commented 5 years ago

https://github.com/flashmob/go-guerrilla/tree/master/backends implemented multiple backends using a Backend interface just as you did with your https://github.com/emersion/go-smtp#server

foxcpp commented 5 years ago

What's up with https://github.com/emersion/go-imap-disk? Can we use it? (of course, we need to finish it first)

emersion commented 5 years ago

We can use it, but it doesn't use maildir, it uses a bolt database. This is fine too, but a maildir backend would probably be higher priority for compatibility (makes migrating easier).

foxcpp commented 5 years ago

https://github.com/luksen/maildir Except that we also need some way to store UIDs and some other meta-data (custom flags, internal date).

lupine commented 5 years ago

Dovecot stores UIDs in a uidlist file. I've been working on a project similar to maddy, on and off (maddy is better!), and implemented a parser for it here: https://gitlab.com/lupine/crockery/tree/master/internal/imap/uidlist

foxcpp commented 5 years ago

We currently want to implement support for Maildir storage by extending https://github.com/emersion/go-imap-disk which uses BoltDB for all meta-data. Personally, I don't like the idea of supporting other servers internal formats in maddy. Also we can't use your code anyway due to incompatible licenses (crockery's AGPL and maddy's MIT).

emersion commented 5 years ago

Are the UIDs the only metadata we need to save?

foxcpp commented 5 years ago

Mailbox-related: UIDVALIDITY, UIDNEXT Message-related: flags, RFC822 size [1], internal date [2], UID

[1] - Probably we would have to store message using native line separators to reduce possible interoperability problems [research needed], but we should still report RFC822 size with CRLF (Is that right?). [2] - Probably FS "modification time"?

emersion commented 5 years ago

Probably we would have to store message using native line separators to reduce possible interoperability problems

No, MIME says you MUST use CRLF. Using non-CRLF line endings with MIME messages leads to various issues including DKIM signature breakage.

foxcpp commented 5 years ago

Related discussion: https://dovecot.org/list/dovecot/2013-May/090499.html

The original (djb) definition of Maildir assumed that messages would be written to the maildir with LF line endings, and both MTA and POP server would translate back to CRLF as needed. Dovecot (as a POP server) can deal with messages in either format, and should always return them to clients with CRLF.

It mentions POP, but I think it applies to IMAP too.

emersion commented 5 years ago

Gah that is annoying.

Also note that:

mail_save_crlf setting controls if mails are saved as CRLF or LF to storage.

foxcpp commented 4 years ago

Started some work here. https://github.com/foxcpp/go-imap-maildir

Not high priority though, go-imap-sql is working reasonably well.

foxcpp commented 4 years ago

go-imap-maildy is ready for use. Integration is blocked by #259 since it was written for v2.

foxcpp commented 2 years ago

Blocked by https://github.com/foxcpp/go-imap-maildir/issues/1 since we need to use go-imap-mess "external update" adapter for maddyctl synchronization.

iacore commented 2 months ago

There is this better idea that improves upon maildir called m2dir: https://bitfehler.srht.site/posts/2024-04-15_m2dir-treating-mails-as-files-without-going-crazy.html

foxcpp commented 1 month ago

go-imap-maildir probably could be refactored to actually support both formats.

foxcpp commented 1 month ago

However, currently the priority is the development of the new version of imapsql (internally named maddy-storage). So unless someone steps up, maildir or m2dir won't be supported for the foreseeable future.