Closed tjltjl closed 1 year ago
(Talking about numbers.Number, naturally)
Thank you for reporting this!
It looks like this is mostly a typeshed issue. Structursl subtyping (see PR #3132) will help, but it looks like still some changes will be necessary in typeshed, since we can't declare Number
a protocol (it is too trivial currently).
@JelleZijlstra what do you think?
Not sure we need to keep this open, if it can be fixed in typeshed once protocols land. (Maybe there should be an omnibus issue for things we can change once it lands?)
Also see PEP 484, which has this to say:
PEP 3141 defines Python's numeric tower, and the stdlib module numbers implements the corresponding ABCs ( Number , Complex , Real , Rational and Integral ). There are some issues with these ABCs, but the built-in concrete numeric classes complex , float and int are ubiquitous (especially the latter two :-).
Rather than requiring that users write import numbers and then use numbers.Float etc., this PEP proposes a straightforward shortcut that is almost as effective: when an argument is annotated as having type float , an argument of type int is acceptable; similar, for an argument annotated as having type complex , arguments of type float or int are acceptable. This does not handle classes implementing the corresponding ABCs or the fractions.Fraction class, but we believe those use cases are exceedingly rare.
I am not too familiar with the numeric tower, but maybe we should just go with PEP 484's recommendation and avoid the numbers ABCs.
numbers.Number
in particular doesn't seem very useful since it declares no members, so logically mypy shouldn't allow you to do anything with a variable declared as a Number
. It looks like the others can be made into Protocols easily when that lands.
What are concrete use cases where one would use the numbers ABCs instead of int/float/complex? Maybe numpy numeric types?
What are concrete use cases where one would use the numbers ABCs instead of int/float/complex? Maybe numpy numeric types?
It's come up a few times before, the only places I found that were using it were just trying to be hyper-correct. E.g. numpy supports it but barely mentions it in its docs.
[Warning: Luddite mode on] IMO PEP 3141 was a flight of fancy and has never produced much useful effect. Pythoneers are generally to close to practice to care about the issue, most of the time the code just works through the magic of duck typing, and when it doesn't people typically say "well that's too fancy anyways". :-)
I'm not sure if the other ABCs beyond Number
are very straightforward either. It would be an interesting exercise for somebody to write a prototype stub (using the protocol PR) that only supports one operation (such as addition) for Complex
, Real
and friends. It would have to work with int
, float
and user-defined classes -- and probably also with mixtures of those when it makes sense.
However, I don't believe that the numeric tower is going to be very useful for type checking. The current types seem to be good enough for the vast majority of use cases, and just providing another set of types for essentially the same purpose seems like unnecessary complexity. In my opinion, a much more promising (and somewhat related) project would be figuring out how to type check code using numpy.
In my opinion, a much more promising (and somewhat related) project would be figuring out how to type check code using numpy.
I agree this looks very interesting. This would require some simple dependent types, since fixed size arrays/matrices are everywhere in mypy. The rules for those are non-trivial, e.g. matrix[n, m] * matrix[m, k] == matrix[n, k]
. But I think fixed size arrays will be quite useful on its own even outside numpy.
It would be really cool at least to document it a bit more clearly than the PEP 484 quote, which I read as
"You don't need to use the numbers.* classes but you can"
when it appears that the real situation is
"The numbers.* classes do not work with typing".
I (quite reasonably?) thought that isinstance(x, cls)
implies that you can have the
x : cls
type declaration.
Are there other places where this is not the case?
(or even just a warning in mypy if one of the numbers.*
types is used?)
Just noting that I seem to have a use-case affected by this; I'm adding annotations for use w/ mypy
to some code that takes latitude/longitude values; so,
def foo(lat, lng):
# type: (?, ?) -> None
I started with float
, but passing an integer is okay. That led me to Union[float, int]
, and subsequently #2128; however, Decimal
is also okay — ideally, I just work in whatever types get input to the function and let duck typing do its thing, but Union[float, int, Decimal…]
is not what I really want to say. So, numbers.Real
, and that brings one to this bug.
(I agree with tjltjl's reasoning that if the specification is x: Cls
, then passing a foo
, where isinstance(foo, Cls)
, should work.)
numbers.Real
doesn't require any special actions, it will be fixed automatically when protocols PR lands (unlike numbers.Number
that also might require some changes in typeshed).
@ilevkivskyi, has that PR landed? Because I have just encountered the same problem with Mypy 0.590:
def bug(n: Real = 1) -> Real:
return n + 1
# error: Incompatible default for argument "n" (default has type "int", argument has type "Real")
Motivating example is HypothesisWorks/hypothesis#200; we have several test strategies for some type of number within a range, and the bounding values can be specified as int/float/fraction - so it would be really nice to show Real
instead of that union!
Yes, but those types are still not labeled as protocols. I personally don't see any harm in labelling them as protocols. Note that the other
type is Any
for all methods, but I don't think this will introduce any additional unsafety as compared to current situation. The problem however is that Number
can't be made a protocol (because it is empty), but it is a base class for all other numbers and protocols can't inherit from non-protocols, so at least for now they stay as ABCs, i.e. this will need a .register()
support, see https://github.com/python/mypy/issues/2922
One consideration around the numeric tower is that when a function returns numpy.float, it would be nice to mark it as returning numbers.Real, since
(1) I don't want to force people to fix type annotations because they happened to return a float in the future (2) You could see both numpy.float and python float returned from the same function for convenience.
Raised priority to high since this is constantly coming.
What solution do you propose?
One possible option is to add a bit of lie to typeshed, see https://github.com/python/typeshed/pull/3108.
Another option is to reject the use of Number
etc. as types (and maybe give a link to documentation that explains why this isn't supported and what to do instead).
Another option is to reject the use of
Number
etc. as types (and maybe give a link to documentation that explains why this isn't supported and what to do instead).
This would be a bit sad, especially taking into account this is a 12-th most liked issue on the tracker. TBH I forgot the details of the discussion, but if the typeshed attempt will work out are there any reasons not to do this?
but if the typeshed attempt will work out are there any reasons not to do this?
I'm wondering if the types will be useful for their intended purpose. If not, the effort may be wasted.
Doesn't seem like this will ever get fixed. Can you at least add a note to the user doc somewhere?
Personally I'm not fussed that (mypy thinks) int
is not a Number
, because I don't have any use for Number
. I am inconvenienced by the fact that (mypy thinks) float
and int
are not Real
, though, since I now have a case where I need to use Union[Real, float]
to satisfy mypy.
Doesn't seem like this will ever get fixed. Can you at least add a note to the user doc somewhere?
Perhaps somebody would like to contribute an item about this to the common issues section in the documentation?
Is it possible, one day, to make a MyPy plugin that can answer the question: "is float a subtype of Real?" Is it possible for plugins to be aware of ABC subtype registration?
This pattern of registering subtypes of ABCs comes up every now and then. In JAX, they have a similar registration mechanism. It would be amazing to have a JAX PyTree type that understood all of the subtypes that are registered.
For now, I have to choose between using an alias to All
and using casts.
I assume that this is also the same root issue:
import numbers
from typing import Union
FixedRational = Union[int, numbers.Rational]
def broken(n: numbers.Rational) -> int:
return n.numerator * n.denominator
def fixed(n: FixedRational) -> int:
return n.numerator * n.denominator
print(broken(3))
print(fixed(3))
Which produces: error: Argument 1 to "broken" has incompatible type "int"; expected "Rational" [arg-type]
I was trying to create a modified Fraction class customized to my needs, and the numbers module is fairly important to that effort. I suppose I can manually create the required Unions, but I'm not sure I understand why this is hard to fix...
Stumbled into this right now, annotating a dataclass that can perfectly hold a float
or a Fraction
: a musical interval which can be both: a rational when we’re lucky—and then it should show to the user as a fraction—and a float when we’re not as lucky, as with almost every regular temperament. Both cases arise more or less equally often.
I suppose I can alias Union[int, Fraction, float]
for a while but it would be really nice if this gets resolved in some manner. :sun_with_face:
This could be solved in typeshed. Right now, types like int and float don't inherit from numbers.Integral and numbers.Real -- I don't recall if there's a technicality involved or whether it's just work (or possibly fear of slowing down everything). At runtime we "register" these virtual subclassing relationships, but static type checkers don't handle ABC.register().
PS. PEP 484 and mypy do treat int as a virtual subclass of float, so you don't need Union[int, float]
.
The technicality I remember being discussed was https://github.com/python/typeshed/issues/3195 We could also make numbers.Real and numbers.Integral (but not numbers.Number) protocols.
Hm, would it be so terrible if numbers.Number
in typeshed didn't have __floor__
and __ceil__
? Or to pretend float
has them? Also, probably few would care if we made Number
a protocol with the same methods as Complex
-- or at least the intersection of Complex and Decimal -- it seems Decimal is the one numeric type that inherits from Number but not from Complex.
PS. PEP 484 and mypy do treat int as a virtual subclass of float, so you don't need
Union[int, float]
.
(Oh. Right, thanks! I even read about this thing in this same issue yesterday before posting, but for some reason forgot that entirely when started writing. :thinking:)
I pushed back on adding these methods to float
, since the runtime doesn't have them. It has been my experience that whenever we cheat in typeshed, it comes back to haunt us later. I don't have a particular strong case here, though, especially since this problem was fixed with Python 3.9.
On Fri, Sep 18, 2020 at 7:33 PM Guido van Rossum notifications@github.com wrote:
Also, probably few would care if we made Number a protocol with the same methods as Complex -- or at least the intersection of Complex and Decimal -- it seems Decimal is the one numeric type that inherits from Number but not from Complex.
This is a great suggestion. I am literally getting ready to suggest a revamp of the numbers module in the standard library—or maybe recreate numeric ABCs elsewhere but adding methods to numbers.Number so that it can be used in type hints.
I've been discussing this with my friend Leonardo Rochael—a tech reviewer of my book. A couple of weeks ago he suggested that the numeric tower could be improved instead of ignored or deprecated. Leo suggested that numbers.Number should declare the common subset of operations available in decimal.Decimal and number.Real.
Since these methods would be added, and the numbers ABCs currently do not implement subclasshook there would be no backwards-incompatible problem.
As a language evangelist and writer, I see the current situation as problematic: we offer the numbers module in the standard library, and then deep down in PEP 484 we tell people not to use it. We should either add a warning to the numbers module docs to steer users away from it or make it work with type hints. I prefer the latter.
I will start a discussion about this in the typing-sig mailing list, because it's really not an issue restricted to mypy.
Cheers,
Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
It's worth noting that there are more potential problems than just figuring out which methods to put in each protocol: it's also hard to come up with the right types. See the code review in #4401 (e.g., https://github.com/python/typeshed/pull/4401#discussion_r468309452) for a taste of the kind of things we'd run into.
I do agree that it's unfortunate that the numbers
module exists but is not useful for static typing.
I am looking forward to Luciano's proposal. This is a situation that deserves to be fixed in the 3.10-3.12 timeframe.
Would it be possible to resolve the situation from #4401 (e.g. int+float -> float) using judicious overloads? Surely adding two Reals shouldn't return a Complex.
I regret to inform my proposal to solve this issue will need to wait until I deliver the manuscript of Fluent Python, 2nd edition to O'Reilly.
I look forward to coming back and collaborating with you all to find a solution—or to celebrate the fix if it's done before I am back.
Just to note that the existing software for symbolic computation based on Python (like Sympy and Sagemath) would really need this kind of abstract numeric types and already use the numbers. ABC to register their own numeric types. Too bad that the usage of numbers. ABC is currently not good for typing annotations. Some annotations were already introduced in Sagemath for experimental purposes, but mypy does not accept them.
Unfortunately, PEP 3141 was perhaps ahead of its time and it doesn't really line up with the realities of Python. If you have serious use cases I recommend just defining your own Protocols and that way you'll get exactly the behaviour you expect.
That said, I'm not necessarily opposed to lying in typeshed (as suggested in https://github.com/python/mypy/issues/3186#issuecomment-695114671 and other comments), so I took a look at how much mendacity / typeshed changes it would take to get the following to typecheck:
from numbers import *
def check_int(i: int) -> None:
a: Number = i
b: Complex = i
c: Real = i
d: Rational = i
e: Integral = i
def check_float(f: float) -> None:
a: Number = f
b: Complex = f
c: Real = f
def check_complex(c: complex) -> None:
a: Number = c
b: Complex = c
Make everything a Protocol.
None of int, float or complex define __complex__
, so you'd need to remove __complex__
from Complex (and subclasses), even though __complex__
is marked abstractmethod. We could also lie the other way, and add __complex__
to int, float, complex — my preference is to try and contain the mistruths within numbers.pyi
, though.
float doesn't satisfy (as currently annotated) Real's def __floordiv__(self, other: Any) -> int: ...
and def __rfloordiv__(self, other: Any) -> int: ...
. I think we could get away with using a self type or returning Real here (not even sure that that would be a lie).
On Python < 3.9, we run into https://github.com/python/typeshed/issues/3195. We'd have to lie and pretend float has __floor__
and __ceil__
on all Python versions.
I haven't looked at positives, just getting negatives. E.g., to get positives for Number
you'll want to look into intersecting Decimal
and Complex
as previously suggested.
The current numbers
stubs are very Any
-happy. Trying to pin those down could expose more issues, as mentioned in https://github.com/python/mypy/issues/3186#issuecomment-696218897
None of int, float or complex define complex, so you'd need to remove complex from Complex (and subclasses), even though complex is marked abstractmethod. We could also lie the other way, and add complex to int, float, complex — my preference is to try and contain the mistruths within numbers.pyi, though.
Why would it be a mistruth to add __complex__
to int
and float
? After all int
already has __float__
.
float doesn't satisfy Real's def floordiv(self, other: Any) -> int: ... and def rfloordiv(self, other: Any) -> int: .... I think we could get away with using a self type or returning Real here (not even sure that that would be a lie).
Shouldn't the type annotations of __floordiv__
in the ABC be fixed? If you lie here, people are going to get into trouble down the road. It looks like the result should be the coerced type as per PEP 238.
>>> (5).__float__
<method-wrapper '__float__' of int object at 0x100c669b0>
>>> (5).__complex__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__complex__'
>>> (3.0).__complex__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'float' object has no attribute '__complex__'
>>> (1+4j).__complex__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'complex' object has no attribute '__complex__'
>>> (5.0).__floordiv__(2.0)
2.0
>>> (5.0).__rfloordiv__(2.0)
0.0
@hauntsaninja Oh I see what you meant by "mistruth" now. Yeah, that bothered me too, so earlier today I posted this to python-ideas. What do you think of the idea to add __complex__
to all Real
classes?
If int, float, complex picked up __complex__
it would mean typeshed would have to lie less, so it's good from a stubs point of view. But in general, modifying the runtime has its own set of concerns. E.g., see veky's comment in https://bugs.python.org/issue38629 (from when __ceil__
and __floor__
got added to float
).
See also a thread started by Luciano Ramalho on typing-sig (and IIRC also python-dev or python-ideas) on this issue.
For anyone looking for it, I guess this is the thread that @gvanrossum is talking about. There's very informative discussion there.
@hauntsaninja That's the perfect comment for this situation! Which option was decided finally in that case? (My preference is "implement dunders that do the right thing…" because you never know when someone will want to generically pass the function cls.__float__
to a functor.) I like the way you put it "it would mean typeshed would have to lie less".
I'm probably punching above my weight class here, but I have spun my wheels to the point that I am now high-sided and need a tow. I suspect I am not the first to find myself in this state, but I am struggling to find any worthwhile guidance.
Python has:
isinstance
checks compared to other mechanisms) as well as an incomplete set of data model interoperability definitionsI want to publish a library that does math on numeric primitives. I want to provide algorithms. I want my customers to be able to supply their own number implementations. I want to support typing annotations so that my customers can use existing tools to know whether their supplied number primitives satisfy interoperability requirements of my library. I also want to be able to check this at runtime in my library. I want to do this:
# fancylib_now_with_numeric_interoperability.pyi
from typing import Iterable
…
def do_some_fancy_thing(lots_and_lots_of_reals: Iterable[SomeRealThing???], some_rational: SomeRationalThing???) -> SomeRealThing???:
…
for real in lots_and_lots_of_reals:
if isinstance(real, SomeIntegralThing???):
optimized_thing()
elif isinstance(real, SomeRationalThing???):
other_optimized_thing()
else:
expensive_but_correct_for_the_general_case_thing()
…
Question: What the heck do I put in for those type annotations or isinstance
checks? My sense is that anyone who's actually tried to shoehorn something into similar spots using the numeric tower and typing has fought in vain against niggling inconsistencies that may each appear like small bricks to be smashed with some effort, but that together form a wall over which it is difficult to climb (and I don't know whether anyone—much less who—has successfully scaled that edifice).
Assertion one: This use case should not be controversial. There are real world applications that depend on this (or at least that are substantially frustrated by not having it). I should be able to author something that users can drop right into SageMath and start using without me having to worry at all about the details of (e.g.) sage.rings.rational.Rational
¹ or having a bunch of special cases around decimal.Decimal
. The standard library should provide me with ergonomic APIs and clear guidance on how to coordinate with numeric primitive authors I've never met or talked to.
Assertion two: There should be a (preferably one) clear way to do this and surfacing that way should be treated with some urgency. It should not require all who come to this problem to exhaust their bandage supplies dressing wounds from contact with sharp edges. It should not require anyone to piece together the reality that ain't as rosy as the glossy marketing materials from various bug reports, rejected patches², forum posts, etc. tucked away in tall towers in various corners of the interwebs spanning over a decade. The prescribed way should live in a prominent place in the standard library. Clear warnings should be placed in close proximity to areas likely to lead one astray from that path. The fact that this issue is so popular should strongly signal that people can't find the guidance (or perhaps even the mechanisms) they need.
EDIT: Requiring each library developer to independently and repeatedly "learn and adapt to the [exact same] situation [at the nexus of numbers and typing]" is precisely the problem I'm attempting to call into focus. That's a bug, not a feature, and a costly one at that. My beef is not that one can't solve this outside the standard library. (There are always solutions.) My beef is that the standard library provides a false sense of security to newcomers and leaves pretty much everyone to learn the hard way that they have to roll their own work-arounds. Unsurprisingly, those "solutions" tend to serve each author's specific needs and are rarely (if ever) useful beyond those contexts. To my knowledge, no one is working toward a generalized solution. All of this adds tremendous friction to achieving interoperability promises as advertised.
Assertion three: Any canonical method should support performant runtime type-checking. Right now (based on my casual measurement), isinstance(…, numbers.Integral)
is about five times slower than isinstance(…, int)
, and that's probably fine. But runtime_checkable
Protocol
s are currently O(n) (where n is the number of methods to check) for isinstance
checks. (This has been raised before.) Most numeric implementations are immutable (as they should be). When you stack up tens of thousands of numeric operations (or more), instance creation, and runtime type-checking with Protocol
s that define one or two dozen methods, you're in for a bad time. I'm nervous performance isn't on the radars of those who are nudging numeric type-checking in this direction at the potential demise of the numeric tower.
Python enjoys a reputation not only among programmers, but mathematicians and scientists for being not only a viable, but sometimes superior alternative to things like R, Mathematica, etc. An absence of clear, generalized guidance on how to address these issues frustrates tool authors' abilities to properly meet the needs of those audiences.
I don't even know where to post this observation/request/advice/rant. It's not clear to me that there's even an owner for this problem. Maybe the Python mailing list would be more appropriate, but that's one of those tall towers, and I think this deserves more visibility. I also think this issue belies the true root cause that is much larger than this particular description of one symptom. Apologies if this is not the right home or an annoyance for some. I suspect it will garner nods from others. Take it for what it's worth, which is probably close to what you paid for it. In any event, I am grateful for any time and attention paid.
¹ Warts aside, but those are implementation problems and should be surfaced to SageMath owners or my library customers through proper type checking.
² As an aside, Kirpichev's questions seem perfectly reasonable to me, but they remain unanswered. That leaves those in my position to frustratingly wonder if—much less where—answers exist. The fact that Kirpichev even made an attempt doomed from the start to be blocked by gatekeepers strongly suggests a lot of fog around that and similar issues like those I raise above. The standard library's now 13+ year refusal to provide an implementation for Decimal(…) + float
seems at odds with the standard library's advice to authors to do precisely that with their own implementations.
Don't waste your time pushing on a rope. Python and tools like mypy may eventually evolve to satisfy all of your needs, but that could take many years. If you want to contribute to that progress, you should think on that scale.
Successful tool and library developers for Python learn to adapt to the situation on the ground.
@posita I'm facing a very similar issue with a library I'm writing that is built on top of mpmath. After struggling for a day or two, this is the (far from ideal) solution I arrived at: https://github.com/jorenham/hall/blob/v0.1.13/hall/numbers.py
Although the library is still very young, bound to change a lot, and probably buggy, perhaps this could give you some ideas on how to solve it for your project :)
I do not in any way wish to complain (I'm extremely grateful for MyPy, and all the work the maintainers put into the project!), but would like to humbly submit that it has taken me >2,000 words and nearly 50 links to explain on Stack Overflow why numbers.Number
is not a valid type hint for numeric types. I'm still not convinced I have all corner-cases covered when it comes to suggesting alternatives.
The status quo is... not user-friendly ☹️
[I]t has taken me >2,000 words and nearly 50 links to explain on Stack Overflow why
numbers.Number
is not a valid type hint for numeric types. …
That is a fantastic post that should never need an audience.
For those landing here in search of something to help bridge the distance now, I have attempted to generalize my own work-arounds for the various sharp edges between numbers and typing in Python. Those work-arounds now live in numerary
(having extracted it via c-section from dyce
, where it was conceived).
Just a heads up that it is constructed on a foundation of deliberate compromises that side-step problems that developers have today. In other words, it's scrappy, not principled. My hope it will be short-lived. Docs are online. It should be considered experimental, but it is rapidly approaching stability. Feedback, suggestions, and contributions are desperately appreciated.
Also hit by not being able to use Number
. I'm trying to create stubs for param which is a bit like dataclass, attrs, pydantic etc. Just very useful for creating graphical user interfaces like the Panel.
param provides a parameter type Number
which validates to any numbers.Number
. So thought I could use numbers.Number
as the type hint, which I cannot.
A nice illustration of what works and does not work is
from numbers import Number
import numpy as np
class MyClass():
value1: Number=1
value2: Number=2.0
value3: Number=np.float64(3.0)
my_class=MyClass()
assert isinstance(my_class.value1, Number)
assert isinstance(my_class.value2, Number)
assert isinstance(my_class.value3, Number)
assert issubclass(int, Number)
assert issubclass(float, Number)
assert issubclass(np.float64, Number)
python script.py
raises no ValidationErrors. But
$ mypy 'script2.py'
script2.py:5: error: Incompatible types in assignment (expression has type "int", variable has type "Number")
script2.py:6: error: Incompatible types in assignment (expression has type "float", variable has type "Number")
script2.py:7: error: Incompatible types in assignment (expression has type "floating[_64Bit]", variable has type "Number")
I would really like some basic Number
type to use for param
and (data) science use cases.
n : Number = 5
producestest_compiler.py:18: error: Incompatible types in assignment (expression has type "int", variable has type "Number")
Which it probably shouldn't because isinstance(n, Number) == True