leycec
commented
3 years ago Bravo! A deep dive on the callable() builtin goes shockingly deep on stdlib internals. This is a lot to masticate my toothless gums on. Let's start with...
Curiosity and Google led to a StackOverflow post with a comment declaring the best way check is EAFP.
EAFP is one of two historical trends in Python that I'd love to see erased into an alternate timeline. The other, of course, is duck typing. Thanks to MyPy & Friends, ...this is us duck typing is going away. That just leaves EAFP.
EAFP is bad because it's slow, because exception handling in Python is slow. Interestingly, it's technically only the except ...: branch of EAFP that's slow; the try: branch is significantly faster, because it's only the handling part of exception handling that's slow. Unless an exception is actually raised, the try: branch only incurs minimal overhead.
Most Pythonistas don't care about performance – or publicly claim they don't, anyway. We do, because performance is our only raison d'être. This means we only EAFP when we either:
- Have no other choice.
- Expect the exact EAFP in question to only rarely (ideally, never) enter the
except ...:branch.
Of course, sometimes we have no other choice. Testing whether an arbitrary module is importable or whether an arbitrary object is hashable are the canonical examples. Thanks to the existence of the __hash__() dunder method that can be overridden to raise arbitrary exceptions, the only robust means of implementing the latter test is:
def is_object_hashable(obj: object) -> bool:
try:
hash(obj)
return True
except:
return FalseWhat can you do, you know?
Note:
inspect.getfileis heavily referenced throughout. It does the heavy lifting ofinspect.getsourcefile
Excellent, my dark acolyte. Excellent. Let us cut out the perfidious middle man! With respect to get_callable_filename_or_placeholder(), we only care about inspect.getfile(). Good catch, it is.
We'll probably never want inspect.getsourcefile() here, because that would imply we are doing something perfidious with source code, which we eventually intend to do (e.g., vile and meddlesome AST transformations) in a completely separate package that is not beartype. But... I digress.
As noted by in the official documentation,
callablecan throw false positives.
Is this still true? Fascinating and horrible if so. We suspect this is only true for callable classes overriding the __call__() dunder method, which we are currently comfortable with ignoring, because we don't even support callable classes yet.
Ignoring callable classes, the callable() builtin should absolutely be reliable. Should. Unless there's black and eldritch magic afoot in CPython's C implementation that I'm blissfully unaware of, which there probably is.
Whats a false-positive? According to
inspect.getfile, function/method objects lacking__code__.co_filenameand classes whose__module__has a__file__evaluate toNone.
Just to be unambiguous, but those edge cases aren't actually false positives with respect to whether or not those objects are callable; those objects are still callable, so there's no false positive. They're just in-memory rather than on-disk, which is absolutely okay and rather common in the dynamic hellscape of interpreted Python.
Wrapper functions generated by the @beartype decorator are a sane example. They're callable functions (which is good), but currently lack __code__.co_filename attributes (which is bad), because they're defined in-memory. In fact, solving the latter issue is exactly why the get_callable_filename_or_placeholder() getter was scoped out in the first place! We have now gone recursive.
The Forest Rather Than the Trees View™ of this whole thing is that the traceback for exception messages raised by wrapper functions generated by the @beartype decorator is currently non-human-readable. Those functions are listed as residing in the non-existing <string> file (which is bad). Moreover, the line number responsible for raising that traceback's exception is listed as the non-existing ??? line (which is even worse).
Solving the former issue requires getting and copying the __code__.co_filename attribute from the decorated callable onto the wrapper function generated by the @beartype for that callable. Enter get_callable_filename_or_placeholder() stage left, folks.
Solving the latter issue requires considerably more care. If my failing memory serves, we need to:
- Synthesize a fake in-memory private
beartypemodule – possibly a unique module for each wrapper function generated by the@beartypedecorator, but hopefully not. - Add each wrapper function to that module.
- Register that module with the stdlib
linecachemodule, updated on each addition to that module.
Behold! It is brutally ugly, but it's not our fault. This is what I tell myself.
Note: Functions/Methods aren't the only objects containing
co_filename.inspect.getfileincludesFrameandTracebackobjects which beartype would omit as they are not callable.
Right. Super catch, Master Helmsman Heliotrope. You continue to have the helm.
This occurs in
inspect.getsourcefilewhich attempts to confirm the file exists viaos.path.exists. If that evaluates to false it checkslinecache.cachefor the file path. We could follow suit or we could forgo the existential questioning...
Yes. Existential questioning is slow and so is inspect.getsourcefile, so let's not walk that lethargic road of inexorable doom. Again, The Forest Rather Than the Trees View™ of this whole thing is just to generate human-readable tracebacks – a very non-mission-critical feature. Generating perfectly accurate, complete, and comprehensive tracebacks that account for all possible edge cases (however unlikely) would be noice for certain definitions of noice, but never at the expense of... well, anything else that actually matters.
...and return a boolean indicating whether the file extension is a byte-code suffix.
Sure! We could try hard. Or we could be lazy. I suggest we be lazy, because my drooping middle-aged gut is the standard I live by. Let's do absolutely nothing and just accept the value of the co_filename attribute as is. In 99.9999999% of real-world cases give or take a digit of accuracy, that value will be what you expect: the absolute filename of the .py-suffixed source module defining that callable.
In the unlikely event that's not the case... well, that's tragic and deplorable and makes my grubby toes curl with indignation, but that's hardly the End of the World As We Know It. Heck, it's not even /r/Python-worthy news – and they'll accept anything there!
I was able to find more articles on Python's garbage collection than I ever expected to.
heh
</face_twitch>
Catching them is pretty easy we just need to add an additional check...
Oh, my vigorous and innocent and easily deceived Padawan. To be young again! Seriously, you're awesome. The above statement, however, is not.
The set of the types of all builtin C-based callables is larger than just types.BuiltinFunctionType – significantly larger. Python internally differentiates between an excrutiating number of different builtin C-based callables. I've seen bound method wrappers, unbound class method descriptors, unbound dunder method wrapper descriptors, and unbound non-dunder method descriptors the horrifying likes of which I pray you never personally witness.
This is why the beartype.cave.CallableTypes tuple exists. I suspect you didn't know about that thing, because that thing is currently undocumented outside that module (which is bad). If we really need to differentiate C-based callables from non-C-based callables at some future point, it'd be trivial to define a new beartype.cave.CallableCTypes tuple reducing to just the set of the types of all builtin C-based callables.
Phew.
Well, that escalated quick.
Heliotrop3
Task List
_BeartypeUtilCallableExceptionget_callable_filename_or_placeholder.__code__.co_filename__module__.__file__isNoneIntroductions
Taken from beartype.beartype._util.utilcallable
Note:
inspect.getfileis heavily referenced throughout. It does the heavy lifting ofinspect.getsourcefileDown the Rabbit Hole
As noted by in the official documentation,
callablecan throw false positives. Curiosity and Google led to a StackOverflow post with a comment declaring the best way check is EAFP. As a following comment points out, there was a brief interlude from Python 3.0 to 3.2 wherecallablewas not in Python. It's reintroduction was, according to the devs, a quality of life revert aimed at addressing long-distance calls: calls to objects far from their point of summoning.Note: There's a more recent post pertaining to the state of
callablein Python 3.10 that's worth keeping track of.Design Notes
Detecting False Positives
Whats a false-positive? According to
inspect.getfile:__code__.co_filename__module__has a__file__evaluate toNoneif sys.modules.get(object.__module__).__file__ is None: ...Note: Functions/Methods aren't the only objects containing
co_filename.inspect.getfileincludes Frame and Traceback objects whichbeartypewould omit as they are not callable.Detecting Compiled File
This occurs in
inspect.getsourcefilewhich attempts to confirm the file exists viaos.path.exists. If that evaluates to false it checkslinecache.cachefor the file path. We could follow suit or we could forgo the existential questioning and return a boolean indicating whether the file extension is a byte-code suffix.Dynamically Allocated Callables.
...I have not been able to find how that's determined. I was able to find more articles on Python's garbage collection than I ever expected to. This will probably be one of those facepalm moments.
Dealing with built-ins
Built-in methods and functions count as...well, methods and functions... Which means they pass through a callable check only to cause trouble later on. Why? "Built-in" means compiled from their C source into the underlying Python framework - no .py file to get. Catching them is pretty easy we just need to add an additional check:
This is different than how
inspecthandles built-ins;inspectdetermines whether an object is built-in using deductive reasoning: "The Callable failed to hit a nestedreturn. Therefore, it must be a built-in type".Closing Note
Further investigation is required into the post pertaining to the state of
callablein Python 3.10.