Emails in RainLoop are not displayed

[tetsumaki]

Describe the bug

Hello, emails are not displayed in RainLoop. We can see that RainLoop detects the email (1) in the column, but it is not visible. I have no problem with K-9 Mail.

Steps to reproduce

I use the latest version of RainLoop: 1.17.0 Just click on Inbox and see that nothing appears.

Log files

I use the default docker configuration with very few changes.

Log from RainLoop when i click on Inbox :

imap: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT STARTTLS LOGINDISABLED COMPRESS] IMAP4rev1 Service Ready
imap: TAG1 STARTTLS
imap: TAG1 OK Begin TLS negotiation now
imap: TAG2 CAPABILITY
imap: * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT AUTH=PLAIN AUTH=LOGIN COMPRESS
imap: TAG2 OK CAPABILITY completed
imap: TAG3 AUTHENTICATE PLAIN
imap: +
imap: TAG3 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT I18NLEVEL=1 SORT THREAD=ORDEREDSUBJECT COMPRESS NAMESPACE] AUTHENTICATE completed
imap: TAG4 CAPABILITY
imap: * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT I18NLEVEL=1 SORT THREAD=ORDEREDSUBJECT COMPRESS NAMESPACE
imap: TAG4 OK CAPABILITY completed
imap: TAG5 STATUS "INBOX" (MESSAGES UNSEEN UIDNEXT)
imap: * STATUS INBOX (MESSAGES 1 UNSEEN 1 UIDNEXT 2)
imap: TAG5 OK STATUS completed
imap: TAG6 LOGOUT
imap: * BYE Closing connection
imap: TAG6 OK LOGOUT completed

And no problem from K-9 Mail when i refresh the Inbox folder :

imap: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT STARTTLS LOGINDISABLED COMPRESS] IMAP4rev1 Service Ready
imap: 1 STARTTLS
imap: 1 OK Begin TLS negotiation now
imap: 2 CAPABILITY
imap: * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT AUTH=PLAIN AUTH=LOGIN COMPRESS
imap: 2 OK CAPABILITY completed
imap: 3 AUTHENTICATE PLAIN
imap: +
imap: 3 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT I18NLEVEL=1 SORT THREAD=ORDEREDSUBJECT COMPRESS NAMESPACE] AUTHENTICATE completed
imap: 4 NAMESPACE
imap: * NAMESPACE (("" ".")) NIL NIL
imap: 4 OK NAMESPACE completed
imap: 5 EXAMINE "INBOX"
imap: * FLAGS (\Seen \Answered \Flagged \Deleted \Draft)
imap: * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft \*)] Flags permitted.
imap: * OK [UNSEEN 1] Message 1 is first unseen
imap: * 1 EXISTS
imap: * 1 RECENT
imap: * OK [UIDNEXT 2] Predicted next UID
imap: * OK [UIDVALIDITY 3991729979] UIDs valid
imap: 5 OK [READ-ONLY] EXAMINE completed
imap: 6 UID SEARCH 1:1 NOT DELETED
imap: * SEARCH 1
imap: 6 OK UID SEARCH completed
imap: 7 UID FETCH 1 (UID FLAGS)
imap: * 1 FETCH (UID 1 FLAGS (\Recent))
imap: 7 OK UID FETCH completed

RainLoop screenshot :

Configuration file

## Maddy Mail Server - default configuration file (2022-06-18)
## This is the copy of maddy.conf with changes necessary to run it in Docker.
# Suitable for small-scale deployments. Uses its own format for local users DB,
# should be managed via maddyctl utility.
#
# See tutorials at https://maddy.email for guidance on typical
# configuration changes.

# ----------------------------------------------------------------------------
# Base variables

$(hostname) = test.domain.net
$(primary_domain) = test.domain.net
$(local_domains) = $(primary_domain)

tls file /data/tls/fullchain.pem /data/tls/privkey.pem

# ----------------------------------------------------------------------------
# Local storage & authentication

# pass_table provides local hashed passwords storage for authentication of
# users. It can be configured to use any "table" module, in default
# configuration a table in SQLite DB is used.
# Table can be replaced to use e.g. a file for passwords. Or pass_table module
# can be replaced altogether to use some external source of credentials (e.g.
# PAM, /etc/shadow file).
#
# If table module supports it (sql_table does) - credentials can be managed
# using 'maddyctl creds' command.

auth.pass_table local_authdb {
    table sql_table {
        driver sqlite3
        dsn credentials.db
        table_name passwords
    }
}

# imapsql module stores all indexes and metadata necessary for IMAP using a
# relational database. It is used by IMAP endpoint for mailbox access and
# also by SMTP & Submission endpoints for delivery of local messages.
#
# IMAP accounts, mailboxes and all message metadata can be inspected using
# imap-* subcommands of maddyctl utility.

storage.imapsql local_mailboxes {
    driver sqlite3
    dsn imapsql.db
}

# ----------------------------------------------------------------------------
# SMTP endpoints + message routing

hostname $(hostname)

table.chain local_rewrites {
    optional_step regexp "(.+)\+(.+)@(.+)" "$1@$3"
    optional_step static {
        entry postmaster postmaster@$(primary_domain)
    }
    optional_step file /etc/maddy/aliases
}

msgpipeline local_routing {
    # Insert handling for special-purpose local domains here.
    # e.g.
    # destination lists.example.org {
    #     deliver_to lmtp tcp://127.0.0.1:8024
    # }

    destination postmaster $(local_domains) {
        modify {
            replace_rcpt &local_rewrites
        }

        deliver_to &local_mailboxes
    }

    default_destination {
        reject 550 5.1.1 "User doesn't exist"
    }
}

smtp tcp://0.0.0.0:25 {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections.
        all rate 20 1s
        all concurrency 10
    }

    dmarc yes
    check {
        require_mx_record
        dkim
        spf
    }

    source $(local_domains) {
        reject 501 5.1.8 "Use Submission for outgoing SMTP"
    }
    default_source {
        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            reject 550 5.1.1 "User doesn't exist"
        }
    }
}

submission tls://0.0.0.0:465 tcp://0.0.0.0:587 {
    limits {
        # Up to 50 msgs/sec across any amount of SMTP connections.
        all rate 50 1s
    }

    auth &local_authdb

    source $(local_domains) {
        check {
            authorize_sender {
                prepare_email &local_rewrites
                user_to_email identity
            }
        }

        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            modify {
                dkim $(primary_domain) $(local_domains) default
            }
            deliver_to &remote_queue
        }
    }
    default_source {
        reject 501 5.1.8 "Non-local sender domain"
    }
}

target.remote outbound_delivery {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections
        # for each recipient domain.
        destination rate 20 1s
        destination concurrency 10
    }
    mx_auth {
        dane
        mtasts {
            cache fs
            fs_dir mtasts_cache/
        }
        local_policy {
            min_tls_level encrypted
            min_mx_level none
        }
    }
}

target.queue remote_queue {
    target &outbound_delivery

    autogenerated_msg_domain $(primary_domain)
    bounce {
        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            reject 550 5.0.0 "Refusing to send DSNs to non-local addresses"
        }
    }
}

# ----------------------------------------------------------------------------
# IMAP endpoints

imap tls://0.0.0.0:993 tcp://0.0.0.0:143 {
    debug
    io_debug
    auth &local_authdb
    storage &local_mailboxes
}

Environment information

Alpine Linux 3.16.2 Podman 4.1.0

• maddy version: docker foxcpp/maddy:0.6.2

[RiadVargas]

Facing the same issue with foxcpp/maddy:0.6.2.

I am migrating from Mailu to Maddy and some emails were displayed correctly. From 6,5k only 22 were listed and I couldn't find anything blatant.

Here's IMAP query made by RainLoop:

[22:09:23.028][ccb98281] IMAP[NOTE]: Start connection to "ssl://mail.(redacted):993"
[22:09:23.042][ccb98281] IMAP[NOTE]: Connected (success)
[22:09:23.042][ccb98281] IMAP[DATA]: < * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT AUTH=PLAIN AUTH=LOGIN COMPRESS] IMAP4rev1 Service Ready\r\n
[22:09:23.042][ccb98281] IMAP[DATA]: > TAG1 AUTHENTICATE PLAIN\r\n
[22:09:23.042][ccb98281] IMAP[DATA]: < +\r\n
[22:09:23.042][ccb98281] IMAP[SECURE]: > *******\r\n
[22:09:23.094][ccb98281] IMAP[DATA]: < TAG1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT I18NLEVEL=1 SORT THREAD=ORDEREDSUBJECT COMPRESS NAMESPACE] AUTHENTICATE completed\r\n
[22:09:23.095][ccb98281] IMAP[DATA]: > TAG2 SELECT "INBOX"\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * FLAGS (\Seen \Answered \Flagged \Deleted \Draft)\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft \*)] Flags permitted.\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * OK [UNSEEN 1] Message 1 is first unseen\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * 6593 EXISTS\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * 0 RECENT\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * OK [UIDNEXT 6594] Predicted next UID\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < * OK [UIDVALIDITY 2966996583] UIDs valid\r\n
[22:09:23.106][ccb98281] IMAP[DATA]: < TAG2 OK [READ-WRITE] SELECT completed\r\n
[22:09:23.107][ccb98281] IMAP[DATA]: > TAG3 CAPABILITY\r\n
[22:09:23.107][ccb98281] IMAP[DATA]: < * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT I18NLEVEL=1 SORT THREAD=ORDEREDSUBJECT COMPRESS NAMESPACE\r\n
[22:09:23.107][ccb98281] IMAP[DATA]: < TAG3 OK CAPABILITY completed\r\n
[22:09:23.107][ccb98281] IMAP[DATA]: > TAG4 STATUS "INBOX" (MESSAGES UNSEEN UIDNEXT)\r\n
[22:09:23.108][ccb98281] IMAP[DATA]: < * STATUS INBOX (UIDNEXT 6594 MESSAGES 6593 UNSEEN 6571)\r\n
[22:09:23.108][ccb98281] IMAP[DATA]: < TAG4 OK STATUS completed\r\n
[22:09:23.108][ccb98281] INFO[DATA]: Get Serialized UIDS from cache ("INBOX" / UNDELETED) [count:22]
[22:09:23.108][ccb98281] INFO[DATA]: ThreadsMapSorted/ALL/Limit=50/INBOX/3eedf1a29937afce36227da926d32a73
[22:09:23.108][ccb98281] INFO[DATA]: Get Serialized Thread UIDS from cache ("INBOX" / ALL) [count:17]
[22:09:23.109][ccb98281] IMAP[DATA]: > TAG5 UID FETCH 2259,1984,1775,1457,934,2720,6461,4294,2742,3293,2534,4351,2691,5362,2692,6435,2698 (UID RFC822.SIZE INTERNALDATE FLAGS BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (RETURN-PATH RECEIVED MIME-VERSION MESSAGE-ID CONTENT-TYPE FROM TO CC BCC SENDER REPLY-TO DELIVERED-TO IN-REPLY-TO REFERENCES DATE SUBJECT SENSITIVITY X-MSMAIL-PRIORITY IMPORTANCE X-PRIORITY X-DRAFT-INFO RETURN-RECEIPT-TO DISPOSITION-NOTIFICATION-TO X-CONFIRM-READING-TO AUTHENTICATION-RESULTS X-DKIM-AUTHENTICATION-RESULTS LIST-UNSUBSCRIBE)])\r\n

[foxcpp]

Seems like SEARCH UNDELETED may be returning incomplete results. Not sure why that might be the case, I will take a look at it in the following days.

[RiadVargas]

@foxcpp I think I've found out what is causing the issue, at least for my case. My emails were migrated through imapsync and for some unknown reason unread emails aren't being flagged at all. That explains why some emails appeared and others don't.

Considering they aren't being flagged and SEARCH UNDELETED refers to flags table - that don't even know about the not flagged messages - it won't list anything.

Here's a query example:

SELECT DISTINCT flags.msgId
        FROM flags
        WHERE mboxId = $1
        AND flags.msgId NOT IN (
        SELECT DISTINCT flags.msgId
        FROM flags
        WHERE mboxId = $2
        AND flag = $3

(fwiw: UNDELETED is properly passing the right parameters to the second query: search stmt args: [5 5 \Deleted])

My workaround was to manually add messages to flags table.

Any particular reason we rely on flags table rather msgs in the first query, Max?

[RiadVargas]

Adding to the above, another way to fix it is to slightly change the above query - formatted at go-imap-sql - to something like this:

SELECT DISTINCT msgs.msgId
        FROM msgs
        WHERE mboxId = $1
        AND msgs.msgId NOT IN (
        SELECT DISTINCT flags.msgId
        FROM flags
        WHERE mboxId = $2
        AND flag = $3)

@foxcpp if you are okay with it I can create the PR and create test scenarios to catch it in the future.

[foxcpp]

@foxcpp if you are okay with it I can create the PR and create test scenarios to catch it in the future.

Yeah, that would be appreciated.

Any particular reason we rely on flags table rather msgs in the first query, Max?

Mostly an oversight. Before go-imap v2 migration, all messages had at least \Recent flag and that flag is immutable per IMAP4 spec so it worked fine.

[stellarpower]

I'm on Madddy 0.7.0 (image 4a1b....fe52), and appear still to be seeing this:

https://github.com/the-djmaze/snappymail/issues/1515

More logs on that issue but let me know if I can provide anyhting.

Cheers