Fix equality checks for some types

[serhiy-storchaka]

BPO	37685
Nosy	@ambv, @serhiy-storchaka, @MojoVampire, @vedgar, @pganssle, @miss-islington, @aeros
PRs	python/cpython#14952 python/cpython#14996 python/cpython#15102 python/cpython#15104 python/cpython#15167 python/cpython#17836

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-bug', 'library', '3.9'] title = 'Fix equality checks for some types' updated_at = user = 'https://github.com/serhiy-storchaka' ``` bugs.python.org fields: ```python activity = actor = 'serhiy.storchaka' assignee = 'none' closed = True closed_date = closer = 'serhiy.storchaka' components = ['Library (Lib)'] creation = creator = 'serhiy.storchaka' dependencies = [] files = [] hgrepos = [] issue_num = 37685 keywords = ['patch'] message_count = 15.0 messages = ['348452', '348514', '348516', '348517', '348518', '348522', '348552', '348977', '348978', '348982', '349218', '349219', '349220', '361876', '361912'] nosy_count = 7.0 nosy_names = ['lukasz.langa', 'serhiy.storchaka', 'josh.r', 'veky', 'p-ganssle', 'miss-islington', 'aeros'] pr_nums = ['14952', '14996', '15102', '15104', '15167', '17836'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue37685' versions = ['Python 3.9'] ```

[serhiy-storchaka]

The __eq__ implementation should return NotImplemented instead of False or raising an exception (like AttributeError or TypeError) when it does not support comparison with the other operand's type. It is so for most of implementations in the stdlib, but there are several exceptions. The proposed patch fixes these cases.

[fe5a23f9-4d47-49f8-9fb5-d6fbad5d9e38]

Wat?? Are we heading towards the world where 3.2 == 'something' is NotImplemented? I understand __lt__, of course, but in my opinion Python has always defined equality comparisons between different types. I think this is a huge regression.

[pganssle]

@veky I can't be sure, but I think you may not understand what returning NotImplemented does - this makes comparisons more versatile, not less.

The way it works is that when Python does, for example, x == y, it will first call x.__eq(y) and check the return value of that. If that's True or False, it will return the value, but if it's NotImplemented, it will call y.__eq(x) and return that value, unless the value is NotImplemented. If both comparisons return NotImplemented, it falls back on the default implementation for __eq, which I think is "x is y". For __lt it's different fallback behavior, but the same general idea.

Most of these changes are going from a situation where __lt or __eq was raising an exception or returning False for comparisons where the operation is not defined. This makes the distinction between "I don't know how to compare myself to that object" and "I know how to compare myself to that object and I am not equal to it".

[99ffcaa5-b43b-4e8e-a35e-9c890007b9cd]

@p-ganssle: Yup. If both sides return NotImplemented, __eq and __ne return a result based on an identity comparison; all other rich comparisons raise TypeError in that case. Code is here: https://github.com/python/cpython/blob/3.7/Objects/object.c#L679

As you said, NotImplemented is never actually returned from syntax based comparisons; you only see it if you explicitly call __eq (and related __XX rich comparison methods) directly (usually because you're implementing one method in terms of another).

AFAICT, about the only time you should ever be returning False rather than NotImplemented is when the other operand is an instance of the same class as you (or a subclass), allowing you to say with certainty that you know how to do the comparison (Python already handles the edge case of subclasses for you, by calling the comparison method on the subclass first, even if it's on the right hand side). In any other case, you return NotImplemented, and let the other side determine if they know how to do the comparison.

[99ffcaa5-b43b-4e8e-a35e-9c890007b9cd]

Serhiy: Is there a reason not to use the functools.totalordering decorator on TimerHandle, so you can get rid of \_le/ge/gt rather than fixing them individually? I notice at least one behavioral difference (totalordering's le/ge methods use ==/!= for the fallback comparison, where TimerHandle explicitly calls \_eq) which this would eliminate, along with reducing the code to maintain.

Specifically, the totalordering fallbacks work if the provided operator (e.g. \_lt) works and either side provides a functional __eq/ne, as appropriate (even if the other side *can't* provide the complementary inequality operator, e.g. __gt). TimerHandle's behavior differs; it's all or nothing (so if __lt works, but __eq fails, then the success of __lt doesn't matter, the other side has to do everything).

I'm not saying totalordering is necessarily correct (it does seem a bit odd that A \<= B can return True based on a mix of results, one from A.\_lt, one from B.__eq), but constantly reimplementing these fallback operations has been a source of subtle bugs so often, that I'd rather depend on a single common implementation, and debate tweaks to it there, rather than having it reimplemented slightly differently in a million different places.

I don't think import times for functools should be a problem; looks like at least four asyncio submodules import it, including asyncio.format_helpers (which asyncio.events takes a direct dependency on), so if you're using asyncio.events, you're already importing functools anyway.

[fe5a23f9-4d47-49f8-9fb5-d6fbad5d9e38]

Thanks for the clarification. Yes, now I see how it really works, but I saw some comment about not wanting to backport it to 3.7, because it changes the semantics. Now that I understand the implementation, it seems to me that it really doesn't change the semantics, except in some weird cases like the objects "accidentally" having the sought attributes. But nevermind, I'm fine with that -- I'm just saying that's the reason I thought the change is much greater than it really is.

[aeros]

Serhiy: Is there a reason not to use the functools.totalordering decorator on TimerHandle, so you can get rid of \_le/ge/gt__ rather than fixing them individually?

I strongly agree with this suggestion, unless there's some unique behavior with TimerHandle objects which would prevent this from being viable. IMO, the @total_ordering decorator should be used as much as practically possible. Individually specifying all of the rich comparisons when we don't have to adds a significant maintenance cost.

[serhiy-storchaka]

New changeset 17e52649c0e7e9389f1cc2444a53f059e24e6bca by Serhiy Storchaka in branch 'master': bpo-37685: Fixed comparisons of datetime.timedelta and datetime.timezone. (GH-14996) https://github.com/python/cpython/commit/17e52649c0e7e9389f1cc2444a53f059e24e6bca

[miss-islington]

New changeset dde944f9df8dea28c07935ebd6de06db7e575c12 by Miss Islington (bot) in branch '3.8': bpo-37685: Fixed comparisons of datetime.timedelta and datetime.timezone. (GH-14996) https://github.com/python/cpython/commit/dde944f9df8dea28c07935ebd6de06db7e575c12

[serhiy-storchaka]

New changeset 6ed20e54e4c110e9adcfb70aba85310625e3edb4 by Serhiy Storchaka in branch '3.7': [3.7] bpo-37685: Fixed comparisons of datetime.timedelta and datetime.timezone. (GH-14996) (GH-15104) https://github.com/python/cpython/commit/6ed20e54e4c110e9adcfb70aba85310625e3edb4

[serhiy-storchaka]

New changeset 662db125cddbca1db68116c547c290eb3943d98e by Serhiy Storchaka in branch 'master': bpo-37685: Fixed __eq, __lt etc implementations in some classes. (GH-14952) https://github.com/python/cpython/commit/662db125cddbca1db68116c547c290eb3943d98e

[serhiy-storchaka]

New changeset 7d44e7a4563072d0fad00427b76b94cad61c38ae by Serhiy Storchaka in branch 'master': bpo-37685: Use singletons ALWAYS_EQ and NEVER_EQ in more tests. (GH-15167) https://github.com/python/cpython/commit/7d44e7a4563072d0fad00427b76b94cad61c38ae

[serhiy-storchaka]

Łukasz, are you fine with backporting PR 14952 to 3.8?

[ambv]

Unfortunately, we released 3.8.0, 3.8.1, and 3.8.2rc1 without this change. It seems too late in the release cycle to introduce this change. I'd feel better seeing it in 3.9 only. Sorry that we didn't act in time to include this in 3.8.0.

[serhiy-storchaka]

Thank you Łukasz. I considered this as a bug fix, but was not sure that we should fix these bugs. They were here from the beginning.

Josh, as for using total_ordering, I think it is a different issue. I did not want to change the behavior except fixing bugs (and raising an error or not supporting comparison with objects which support comparison when used at LHS looks like a bug). Feel free to open separate issues for using total_ordering or changing its semantic.