Better handling of invalid Date headers

[p91paul]

I spotted in some emails a date in the following format:

Date: 06-MAY-2016 19:00:22

I think due to the uppercase month, k-9 mail is unable to handle them, and displays them as they were sent at the time they were received. Can you work around this issue?

[p91paul]

Oh, I forgot: I am receiving said emails with IMAP

[cketti]

We won't add support for decoding another non-standard date format. What we could do is mark these messages in the database and display the value of the date header when viewing the message. For sorting purposes we'd still have to use the date of retrieval, though.

[p91paul]

Looks perfectly reasonable to me. You should still evaluate if you could be confusing your users showing a date which is different from the sort order; I would do something like a red date, and if the user taps it shows a popup of some sort explaining what happened with that date. Thanks for considering this!

[dvb15]

Sorry for commenting old issue, but why do you use the client's retrival date as a fallback? If a user sorts incoming mail by creation date (not reception) in k-9, maybe you'd use the closest available recognized date (I mean one in first receive header, the second one if the first is unavailable too and so on)? The client's retrival date can be too far from creation date causing wrong sorts.

[marmistrz]

There's one problem with just ignoring invalid dates. Many (too many to ignore) people use the gmail's function of retrieving messages from different e-mail accounts via POP3 and gmail doesn't sanitize the dates here. So even if the original server fixes the date headers to be RFC-compliant, the mails already fetched are irrecoverably broken.

It'll just end up with non-technical people ranting that gmail just works

[StefanBertels]

I have (probably) similar problem (k9mail shows date of sync instead of correct value). IMHO the raw value should be displayed instead. Regarding sorting order I would prefer to have the order of messages as on (IMAP dovecot) server which is order of arrival (or folder move).

Alpine shows this date as correct: Thu, 21 Nov 2013 21:28:42 GMT-07:00

[maxthiel]

Hi,

I'll comment on here instead of opening a new issue, since my problem appears to be quite similar. I've got some email from a specific sender that are displayed with the time of sync. I don't believe that the Date field is an improper format... since I can't find any Date field in the headers.

The fields I have are:

Return-Path: <xxxxxxxxxxxxxxxxx@senderdomain.com>
Delivered-To: xxxxxxxxxxx@domain.tld
Received: from localhost (HELO queue) (127.0.0.1)
    by localhost with SMTP; 20 Feb 2019 10:26:15 +0200
Received: from unknown (HELO mail.ovh.net) (123.456.789.123)
    by ovh.net with AES256-GCM-SHA384 encrypted SMTP; 20 Feb 2019 10:26:15 +0200
Received: from mail.ovh.net (unknown [10.101.8.47])
    by mail.ovh.net (Postfix) with ESMTP id 4449fv4q1jz25pXfp
    for <xxxxxxxxxxx@domain.tld>; Wed, 20 Feb 2019 08:26:15 +0000 (UTC)
Received: from mail.ovh.net (unknown [10.101.4.36])
    by mail.ovh.net (Postfix) with ESMTP id 4449fr1xWQzd0Ys5
    for <xxxxxxxxxxx@domain.tld>; Wed, 20 Feb 2019 08:26:12 +0000 (UTC)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=123.456.789.123; helo=subdomain.senderdomain.com; envelope-from=e-tradealerts-donotreply@senderdomain.com; receiver=<UNKNOWN>
Authentication-Results: mail.ovh.net;
    dkim=pass (2048-bit key; unprotected) header.d=senderdomain.com header.i=@senderdomain.com header.b="QKvEtMWj";
    dkim-atps=neutral
Received: from subdomain.senderdomain.com (subdomain.senderdomain.com [123.456.789.123])
    by mail.ovh.net (Postfix) with ESMTPS id 4449fq6mXczD02cW
    for <xxxxxxxxxxx@domain.tld>; Wed, 20 Feb 2019 08:26:11 +0000 (UTC)
Received: from Alerts.senderdomain.com (subdomain.senderdomain.com [123.456.789.123])
    by subdomain.senderdomain.com (Postfix) with SMTP id 708AC17954C
    for <xxxxxxxxxxx@domain.tld>; Wed, 20 Feb 2019 03:26:03 -0500 (EST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=senderdomain.com;
   ......
To: xxxxxxxxxxx@domain.tld
From: xxxxxxxxxx@senderdomain.com
Reply-To: xxxxxxxxxx@senderdomain.com
Subject: yyyyyyyyyyy
MIME-Version: 1.0
Content-Type: text/html
Content-Disposition: inline
X-Ovh-Remote: 123.456.789.123 (subdomain.senderdomain.com)
X-Ovh-Tracer-Id: 1090997012543130857
X-VR-SPAMSTATE: OK
X-VR-SPAMSCORE: 49
X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedtledruddtledgjeelucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucfpohcuuggrthgvuchfihgvlhguucdlgeelmd
X-Ovh-Spam-Status: OK
X-Ovh-Spam-Reason: vr: OK; dkim: disabled; spf: disabled
X-Ovh-Message-Type: OK

Roundcube displays the message with the "right" date. Would it be possible for K9 to use a value from one of the Received field, when no date is present, instead of the value from the last sync?

[cketti]

I have changed my mind on this. I'm no longer opposed to performing all kinds of magic to derive a sensible display date from a message. However, in those cases I want K-9 Mail to display an error banner, letting users know that the email is broken. Ideally users will contact the sender of the message and get them to fix their system so future messages won't show up in K-9 Mail with an annoying error banner.

[maxthiel]

Sounds fair to me. Do let me know if you need more details or other example than the one I posted above.

[StefanBertels]

I think many mails with broken/missing date headers are created by poor mail scripts (newsletter / spam mailers) and sometimes the date value is wrong (unrealistic old date / future date), too. IMHO it's totally fine to have some "ugly" (less clean/polished) display style in details view for those (like banner / warning sign ⚠ / additional line etc).

Some thoughts: a) date is correct => internally use and display this parsed value b) date is not correct but can be parsed (fallback logic) => internally use parsed value, but display raw value with warning sign. c) date is not parseable or ambiguous or missing: find some fallback date in "Received:" header (IIRC Outlook does something like this when ordering by arrival) and internally use that value. Display replacement value + warning + raw value (or "Date header missing") + maybe some hint/icon that raw value is a guess/replacement

Just my 2 cent.

[mkende]

One reason I think that the current approach should be changed (trying to use any date from the headers, rather than the received date of the email) is because ideally there is no state in an IMAP client. So, when I see a mail displayed in K-9 mail with the wrong date, I immediately start to fear that it has corrupted my emails. This isn’t the case but still frightening.

I also don’t think that displaying an error banner should be a requirement for such a change: anyway displaying the retrieval date is already a broken situation. Maybe a banner should be shown, or maybe not, but there is no reason that a banner is more required if the app is showing a date extracted from another header, rather than the retrieval date.

[cketti]

I want to show an error banner because silently fixing the situation will take away the little bit of pressure there is to fix the bug causing the problem. Showing a banner is more annoying than showing a wrong date. So ideally users will a) blame the correct party and b) create enough pressure to get the software fixed that generated the broken message.

Adding the "date fixing" first and the error banner later, is a great way to never show an error banner because nobody will ever get around to adding it. And now K-9 Mail is part of the problem (silently fixing the errors of others).

[mkende]

Ok, that’s fair. I just hope that it does not mean that the more complex fix is not implemented ("the best is the enemy of the good" as we say in French).

I would also recommend that the banner is more an "info" than an error, to not scare users who might not understand what this is about.

[njeyaakili]

The origination date aka "Date:" field is a (one of two) standards-required header field. I don't think that a client should try to reverse engineer a "correct" date. Rather I wouldn't display any date and would let the message fall where it may on a "date sent" sort (with a "no date provided" error message in place of the missing date).

There are clear standards (IETF RFCs) for these things. I don't favour mail client developers spending their time trying to patch things up for mail server administrators who can't be bothered with running standards-compliant servers. As cketti noted, if you silently fix these types of errors you become part of the problem (and waste a lot of time in the process).

[mkende]

There are new buggy softwares written everyday, computers with mis-configured date, etc.. So even if they are regularly fixed, a user of K9 mail will continue to receive faulty emails forever. While we can try to improve the situation by displaying a warning and hoping that users will reach out to the sender, providing a voluntarily broken experience would ultimately be punishing K9 mail users for something that is out of their hands (but I understand that this is not what @cketti was proposing).

Also, the Date: header is required in an email, but there is no provision that this is the header that must be displayed by an email client. A simpler approach than the proposed parsing of other header could be to use email the internal date from the server. I believe that this is reasonably common (some clients offer the option to use that one by default).

[waldner]

Could I ask for the (very simple IMHO) option to just leave emails with missing Date: header alone (that is, actually leave blank the spot where the date appears, rather than deduce or synthesizie a date), and sort them as being the oldest in the list?

[tdboorman]

IMHO, emails with a missing or invalid Date: header should use the date from the first Received: header. I would have no objection to a warning banner being displayed on them though.