imaplib: Internaldate2tuple produces wrong result if date is near a DST change

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

BPO	10941
Nosy	@birkenfeld, @abalkin, @bitdancer
Files	imaplib_Internaldate2tuple_dst_fix_python32.patch imaplib_Internaldate2tuple_dst_fix_python27.patch issue10941.diff issue10941_python3.diff issue10941_python2.diff issue10941.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/abalkin' closed_at = created_at = labels = ['type-bug', 'library'] title = 'imaplib: Internaldate2tuple produces wrong result if date is near a DST change' updated_at = user = 'https://bugs.python.org/lavajoe' ``` bugs.python.org fields: ```python activity = actor = 'r.david.murray' assignee = 'belopolsky' closed = True closed_date = closer = 'r.david.murray' components = ['Library (Lib)'] creation = creator = 'lavajoe' dependencies = [] files = ['20446', '20447', '20615', '25285', '25297', '26100'] hgrepos = [] issue_num = 10941 keywords = ['patch'] message_count = 27.0 messages = ['126503', '126573', '127490', '127502', '127507', '127508', '158645', '158646', '158661', '158662', '158663', '158664', '158665', '158733', '158798', '158901', '159105', '159112', '159144', '159146', '159148', '159151', '159649', '163505', '163510', '221896', '221897'] nosy_count = 5.0 nosy_names = ['georg.brandl', 'belopolsky', 'r.david.murray', 'lavajoe', 'python-dev'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue10941' versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4'] ```

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

DST is not handled correctly. Specifically, when the input date/time, ignoring the TZ offset (and treated as if it is local time) is on the other side of the DST changeover from the actual local time represented, the result will be off by one hour. This can happen, e.g., when the input date/time is actually UTC (offset +0000).

This is because the check for DST is done after converting the time into a local time tuple, thereby treating as real local time. This can be corrected by keeping the time in real UTC (by using calendar.timegm() instead of checking the DST flag) until the final conversion to local time.

Here is an example: Run the following two dates, that represent exactly the same time, through Internaldate2tuple:

'25 (INTERNALDATE "01-Apr-2000 19:02:23 -0700")' '101 (INTERNALDATE "02-Apr-2000 02:02:23 +0000")'

Note that a variant of this issue (but identifying it as a different problem) was addressed in a similar way in an old post I found by Colin Brown in 2004 (http://www.velocityreviews.com/forums/t336162-imaplib-function-bug.html).

Patch also adds unit test to check the above example dates. Python 3 version of patch assumes patch from bpo-10939 has been applied.

Patch also corrects code python doc that currently states time is UT, not local (see bpo-10934).

[abalkin]

I agree that this is a bug and the solution is right, I am not sure this should be applied during RC period. The bug has been present and known for a long time, so affected parties probably have a work-around in place already.

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

Here is a new patch that adds a test to the tests committed for bpo-10939.

This new test case is needed to trigger the DST issue. This test is not timezone-dependent (in the sense that one does not have to have a specific timezone set for it to work), but it does depend on the DST change happening at 02:00 on the first Sunday in April, 2000. This is the case for most of the United States, but to make it work everywhere, I have added code that sets the TZ to one that works, and then reverts the temporary TZ change after the test.

The bug is triggered when the time described in the internal date string, ignoring the time offset, is between 02:00 and 03:00. This is because the raw time in the string is interpreted as local time, and local times in this range are basically invalid, since time advances to 03:00 when 02:00 is reached because of the DST change. This causes the final result to be off by one hour.

[This patch is only for Python 3]

[abalkin]

On Sat, Jan 29, 2011 at 4:45 PM, Joe Peterson \report@bugs.python.org\ wrote: ..

I have added code that sets the TZ to one that works, and then reverts the temporary TZ change after the test.

Your test will not restore TZ if it fails. This is not nice. To fix that, code restoring the TZ should go into a finally clause. Even better, set TZ in a setUp() method and restore it in a tearDown() method of the test case class. Ultimately, if you want to be fancy, take a look at the @run_with_locale decorator defined in Lib/test/support.py. We may have more than one use for @run_with_tz.

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

I like the idea of adding the decorator. New patch added.

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

[just carrying over some info from bpo-10939 that is related to this issue]

Here is another manifestation of this issue, related to the local time assumption, but not to DST, per se:

Here is the definition for Europe/London in the unix tz data:

# Zone NAME GMTOFF RULES FORMAT [UNTIL] Zone Europe/London -0:01:15 - LMT 1847 Dec 1 0:00s 0:00 GB-Eire %s 1968 Oct 27 1:00 - BST 1971 Oct 31 2:00u 0:00 GB-Eire %s 1996 0:00 EU GMT/BST

So London's local time was always 1 hour ahead of UTC (BST time) from 1968 Oct 27 until 1971 Oct 31 2:00u.

Because of this, Internaldate2tuple() returns the wrong local time (00:00 rather than [the correct] 01:00) in its tuple when input is "01-Jan-1970 00:00:00 +0000" (the epoch).

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

It's been over a year since any activity on this bug, and it is still in the "patch review" stage. Any news? Anything else I can provide to help to get this moved to the next stage? Thanks!

[bitdancer]

Someone needs to find time to review it.

You could try recruiting reviewers on python-list. Anyone can do a review. Obviously the more knowledge they have in this area the better, but any good review is likely to move the issue along.

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

Ah. I figured that one of the Python devs would be who would review it. Is this the normal path for bugs? Not to question the process, but I find it surprising, since I would think that it would cause a lot of bugs to stall out. Also, I had no idea it was the submitter's responsibility to recruit reviewers. And I am not quite sure how to go about this. I do care about this bug (as it's pretty serious that it returns wrong results), so I am willing to do it. Any wiki page or anything that describes the recruiting process or has more info?

[5579dc13-9f48-42d1-bb17-9c003ef6fa70]

I'll review your patch.

On Apr 18, 2012, at 4:53 PM, Joe Peterson \report@bugs.python.org\ wrote:

Joe Peterson \joe@skyrush.com\ added the comment:

Ah. I figured that one of the Python devs would be who would review it. Is this the normal path for bugs? Not to question the process, but I find it surprising, since I would think that it would cause a lot of bugs to stall out. Also, I had no idea it was the submitter's responsibility to recruit reviewers. And I am not quite sure how to go about this. I do care about this bug (as it's pretty serious that it returns wrong results), so I am willing to do it. Any wiki page or anything that describes the recruiting process or has more info?

----------

---

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue10941\

---

[bitdancer]

I see Alexander is going to take care of this. But to clarify what I suggested for your information:

In an ideal world it would be a committer doing the patch review, followed by a checkin. But in the real world there aren't enough of us with enough time to get to all the bugs with patches. You asked how to move it along and I suggested one way: get someone else to do a review. I wouldn't say that the submitter recruiting a reviewer was a "normal" process, but it is a way to get bugs unstuck. And we get reviews from non-committers frequently (it is a step along the path to becoming a committer...quality reviews are as important as quality patches).

I don't think there's anything in the devguide about this particular nuance.

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

Thanks!! :)

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

David, I understand - thanks for the details. Hopefully I can return the favor and review one in the future.

[abalkin]

Joe,

Your changes to the test suit don't apply cleanly anymore. I can probably fix the conflicts, but if you could post an updated patch it will help.

Thanks.

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

OK, fixed patch to apply cleanly to current code. BTW, this is only for python3. Is it still appropriate to patch python2? And if so, what is the correct code repo to check out for that?

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

I have now included a patch for 2.7. Here are the two latest patches:

Python 2: issue10941_python2.diff Python 3: issue10941_python3.diff

[abalkin]

I still get conflicts:

$ patch -p0 < imaplib_Internaldate2tuple_dst_fix_python32.patch 
patching file Lib/imaplib.py
Hunk #2 FAILED at 1312.
Hunk #3 succeeded at 1347 (offset 8 lines).
Hunk #4 FAILED at 1379.
2 out of 4 hunks FAILED -- saving rejects to file Lib/imaplib.py.rej
patching file Lib/test/test_imaplib.py
Hunk #1 FAILED at 24.
1 out of 1 hunk FAILED -- saving rejects to file Lib/test/test_imaplib.py.rej

Are you sure your patch is against the head of the hg repository?

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

The one you tried is the old patch. Here are the two new patches. Use these:

Python 2: issue10941_python2.diff Python 3: issue10941_python3.diff

[abalkin]

My bad. For some reason I assumed that the latest patch would be at the top of the files list.

David, the patch is good. Should I go ahead and commit?

[29641ad9-fadf-4eb2-b6bc-3b4e6a10e636]

Great to hear, Alexander. Thanks for reviewing! Is it also possible to get the pyhton2.7 version one in?

[abalkin]

On Tue, Apr 24, 2012 at 10:42 AM, Joe Peterson \report@bugs.python.org\ wrote: ..

 Is it also possible to get the pyhton2.7 version one in?

I don't see any reason not to. This is a bug fix and should go into a maintenance release. I will wait to hear from David, who is the maintainer of this module, though.

[bitdancer]

If you are satisfied with the time logic then yes, please apply it. I suspect that the number of people using the code and not aware of the problem (or not caring enough to do anything about it) exceeds the number aware of it who have coded a workaround, so I think putting it in 2.7 (and 3.2) is better than not doing so.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset 90b9781ccb5f by Alexander Belopolsky in branch '3.2': Issue bpo-10941: Fix imaplib.Internaldate2tuple to produce correct result near http://hg.python.org/cpython/rev/90b9781ccb5f

New changeset 6e029b6c142a by Alexander Belopolsky in branch 'default': Issue bpo-10941: Fix imaplib.Internaldate2tuple to produce correct result near http://hg.python.org/cpython/rev/6e029b6c142a

[abalkin]

I updated the fix to take advantage of resolved bpo-9527. I also noticed and fixed another bug: Internaldate2tuple was using locale-dependent %b directive for strftime.

[abalkin]

The latest patch belongs to bpo-11024.

[abalkin]

David,

Is there anything left to do here that is not covered by bpo-11024?

[bitdancer]

Not that I can see.