Aspectize should be more user-friendly with decorating methods

[leycec]

Yay for this! As a terse and insufficient follow-up to beartype/beartype#73, I was thinking of maybe a decorator-via-aspectization justuse API resembling:

from beartype import beartype
from justuse import use

# A "muh_package" proxy transitively decorating all callables
# whose fully-qualified names match "muh_package.*" with
# the @beartype decorator. It feels good, man.
muh_package_beartyped = use(
    'muh_package',
    aspectize_package_callables_with_decorator=beartype,  # <-- heh
)

As parameter names go, aspectize_package_callables_with_decorator is longer than Satan's grubby beard. That's bad. But that's the name we're going with! You previously mentioned pattern matching via an arbitrary boolean tester (e.g., inspect.isfunction()), which suggests you have more general-purpose ideas in mind. That's good.

Let's general-purpose that API up. I leave this in your ingenious Eurozone hands. May we all survive the brewing pandemic hellstorm and live to tell the tale over thick and pungent tiergarten lager.

[leycec]

Oh. You already invented something substantially smarter by overloading the matrix multiplication operator on proxied modules to aspectize an arbitrary decorator across those modules:

import use
from beartype import beartype
from use import isfunction

np = use('numpy')
np @ (isfunction, '', beartype)

Welp... that's a micdrop. :microphone:

[amogorkon]

Hehe.. well, I wished it was all that simple. At the moment we just recycle inspect.isfunction, thus only catch pure functions and only go through the first level with no recursion (thus methods already can't work).

To make this work with beartype we need: 1) make a check that catches everything beartype can be applied to 2) recursively apply the decorator throughout the object hierarchy (probably closures and properties will need extra attention) 3) have a kind of dict of set and deque to prevent applying decorators recursively in a loop and be able to undecorate on the fly (just because we can)

[amogorkon]

@leycec the broadest and most naive check function for beartype I guess would be

def beartypeable(obj):
    try:
        beartype(obj)
        return True
    except:
        return False

maybe we can narrow it down a bit and make it also applicable to users who don't use beartype (boo!) ;-) Any suggestions?

[leycec]

Right. Classic Easier to Ask for forgiveness than Permission (EAFP) design pattern up in here.

As usual, that definitely works – but compromises performance in a probably measurable way, because exception handling is as slow as @leycec doing a double face-palm in front of a double rainbow live on Twitch. Technically, exception handling is fast unless an exception is actually raised. Sadly, that's sorta the same thing in this case, because classes are everywhere and @beartype currently raises exceptions on classes. </gulp>

Given that, isinstance()-based detection would service everyone's need for speed while still working. For his and her maintainability, we go the extra mile and depend purely on builtins to do it. Hu-ah! In my sleep-deprived mind's eye, I envision a devious one-liner that looks like:

from types import FunctionType

def ispythonfunction(obj):
    '''
    ``True`` only if the passed object is a **pure-Python non-class callable**
    (i.e., function or method defined in Python).
    '''
    return isinstance(obj, FunctionType)

# This alias is currently true. Beartype will decorate classes too at some
# future date that shall be heralded for all time, however. At that time,
# this alias will probably need to be replaced with something resembling:
#     def isbeartypeable(obj):
#         return isinstance(obj, (FunctionType, type))
isbeartypeable = ispythonfunction

:exploding_head:

[amogorkon]

After realizing that object.__getattribute__ is the only reliable source of truth regarding attribute discovery, I was able to simplify the aspectizing code quite a bit and make deep recursion work. Check it out :) https://github.com/amogorkon/justuse/blob/5328e253819aa5f3aed86f14aa41a17e5216bb63/src/use/aspectizing.py#L28 And I also learned that everything that indeed IS callable MUST have a (albeit often hidden) .__call__ method.

def f(): pass

class A: pass

a = A()

print(object.__getattribute__(f, "__call__"))
print(object.__getattribute__(A, "__call__"))
print(object.__getattribute__(a, "__call__"))

----
<method-wrapper '__call__' of function object at 0x000001B31733FCA0>
<method-wrapper '__call__' of type object at 0x000001B31900E0D0>
Traceback (most recent call last):
  File "F:\Dropbox (Privat)\code\justuse\tests\.tests\.test3.py", line 9, in <module>
    print(object.__getattribute__(a, "__call__"))
AttributeError: 'A' object has no attribute '__call__'

Could we take advantage of this fact somehow?

[amogorkon]

Also, there's one small issue with closures

def f():
   def g():
       print("foo")

>>> f.g
Traceback (most recent call last):
  File "<pyshell>", line 1, in <module>
AttributeError: 'function' object has no attribute 'g'

g doesn't exist outside of calling f, so there's no way to decorate with our "classic" approach. We need to manipulate the code of f on either the AST or bytecode level to get there.

[amogorkon]

What about properties? methods? classes that act like functions either by __init__ or __call__ .. and now we're wrestling with Enum members that have some __func__ causing recursion like use.Modes.auto_install.fastfail.no_public_installation.auto_install.fastfail .. AHHHH... going crazy

[amogorkon]

inspired by my question on discord #esoteric-python a possible POC for decorating closures: ( https://gist.github.com/thatbirdguythatuknownot/524234dbe8415f1873da57713fa0869c )

import types
import opcode
MAKE_FUNCTION = opcode.opmap['MAKE_FUNCTION']
del opcode

def decorate_inner(decorator, name):
    def actual_decorator(outer_fn):
        # Try and find the decorator function in co_consts,
        # perform EAFP
        co_consts = outer_fn.__code__.co_consts
        try:
            index = co_consts.index(decorator)
        except ValueError:
            index = len(co_consts)
            outer_fn.__code__ = outer_fn.__code__.replace(
                co_consts=co_consts + (decorator,)
            )

        # Try and find the name in co_consts,
        # perform EAFP
        try:
            index2 = co_consts.index(
                f"{outer_fn.__code__.co_name}.<locals>.{name}"
            )
        except ValueError:
            raise RuntimeError(f"couldn't find name {name}") from None

        if index2 > 127:
            raise IndexError(f"name index in co_consts is > 127")

        co_code = outer_fn.__code__.co_code
        character = chr(index2)
        base = co_code.find(f'd{character}'.encode())

        while co_code[base+2] is not MAKE_FUNCTION:
            base = co_code.find(f'd{character}'.encode(), base+2)

        outer_fn.__code__ = outer_fn.__code__.replace(
            co_code=(co_code[:base+4]
                    + f'd{chr(index)}\x02\x00\x83\x01'.encode('l1')
                    + co_code[base+4:])
        )
        return outer_fn
    return actual_decorator

def deco(fn):
    def wrapper(*args, **kwargs):
        print("Calling", fn.__name__)
        ret = fn()
        print("Called", fn.__name__)
        return ret

    return wrapper

@decorate_inner(deco, "g")
def f():
    def g():
        return 42
    return g

g = f()
print(g())

[leycec]

OMFG. You've gone into the Dark Zone – that dimly lit crawlspace of Python that only fearless heroes equipped only with a hand-cranked Guido-brand LED headlight dare to tread. I actually didn't know most of the esoteric plumbling you just uncovered like that one-to-one relation between callables and objects with __call__() dunder methods, so this is equally deeply frightening educating for me. That said...

I honestly feel kinda bad. I think we all glibly assumed aspectization was gonna be easy, right? It's a solved problem. Then real-world edge cases like recursive Enum nastiness and deeply nested callables rear their unwashed purulent scalps. Suddenly, it's all gone grimdark. I'm now genuinely concerned about consuming your precious sanity, volunteer enthusiasm, and free time with an increasingly dicey proposition that may not even be fully decidable or solvable in the general case.

Decorators: Feels Bad Even under the Best Case Scenario

Deeply nested callables only magnify that diceyness.

Before going full tilt on a low-level co_const approach that looks scary and smells bad, I'd probably first have a go at decorator unwrapping. The core insight here is that all sane decorators (which means all decorators anyone should care about) return [wrapper callables (i.e., functions decorated by the standard @functools.wrap decorator)](). Each wrapper callable slyly masquerades as its wrappee callable (i.e., the user-defined callable decorated by that wrapper) by propagating critical metadata like the callable name, docstring, and annotations dictionary from that wrappee onto itself.

Usefully, the @functools.wrap decorator preserves access to the wrappee via the __wrapped__ dunder attribute defined on the wrapper. Since callables are often decorated by multiple decorators, retrieving the original undecorated wrappee callable from a stack of deeply nested wrappers is trivial if and only if all of those wrappers are themselves decorated by the standard @functools.wrap decorator. Of course, all sane decorators leverage @functools.wrap.

Of course, this still doesn't address non-decorator closures or non-closure nested callables. But... both of those seem beyond the mandate of sane aspectization. I'd be especially wary of hard-coding unsafe assumptions about code objects into the steaming guts of justuse. CPython version updates often alter code objects internals, published dunder attributes, and use patterns. Too much bleak wizardry like that could render justuse more brittle and fragile than it would otherwise be.

Of course, a bit of that sort of bleak wizardry is both inevitable and acceptable. For efficiency, @beartype itself now prefers to directly introspect code objects rather than defer to the stable (but really slow) inspect module. Still, we'd have preferred not to do that. We'll have to maintain that in perpetuity against the perennially moving target that is CPython. :dart:

Woe is us who tread these dark waters.

[amogorkon]

Nah, don't feel bad. We've been in much worse situations with justuse, this is actually fun - making visible progress every day. We solved the deep recursion mostly, only Enums seem to defy all logic thus far - but we will bent it to our will in time! We also found a paper on comparing different AOP frameworks for python, so we have some inspiration for the tricky parts. We're in a perfect position for decorating modules, directly at the source, so we'd be foolish to not embrace this opportunity, even if it means to sell small parts of our souls to call on demonic forces.. Decorating closures probably is the most tricky part of it all. The poc manipulating the bytecode definitely wouldn't be my first choice though. I'm guessing manipulating the ast should be much less scary and brittle.

[leycec]

I'm guessing manipulating the ast should be much less scary and brittle.

heh

My one good eye is twitching behind its gilt-encrusted monocle. Although the above statement is absolutely true with respect to bytecode manipulation, AST manipulation itself is still scary and brittle. Notably, AST representations change between Python versions. Of course, this has to be the case; if it wasn't, we couldn't improve the core language with new syntactic primitives. :monocle_face:

This means you usually don't want to directly bang on the output of the core ast module. Instead, you usually want to depend upon a nice intermediary middle-man like Generic AST (GAST). Of course, nobody wants to add yet another dependency to anything. You could always do what we do and pretend that your test suite already catches all possible portability issues. If tox ain't worried, we ain't worried!

[amogorkon]

I could also imagine some possible middleground for closures for the time being. It's much easier to check for closures than changing code on the fly, so we could simply give the user a heads up on unreachable callables and tell them to do it manually. We could always come back to it at a later time..

[leycec]

That's... a stupendous idea, actually. There are just sooo many red-headed level 99 dragons breathing fire across aspectization schemes that depend on fragile bytecode and/or AST manipulation. Emitting non-fatal warnings and/or log messages with the INFO or WARNING levels preserves scarce developer sanity while still enabling end user interventions. Cat @leycec approves. :cat2:

[amogorkon]

After a couple of busy months and fixing other stuff, I'm revisiting aspectizing, intending to finish what I started.

First I'm implementing an overview of everything that's been decorated by using a generated HTML with CSS & Brython. The user should also be able to try different filters to get an idea how to adjust their code to get the desired results. I've successfully realized a similar approach for selecting hashes, where the user can select everything they want and then just click a button to copy&paste a generated snippet ( see https://codepen.io/amogorkon/pen/OJzJgpx ), so I know this will work. This should help greatly with debugging and overall usability, which is critical (scrolling through thousands of log entries is not very enjoyable and productive, at least for me). As usual, feel free to steal the idea ;-)

Then there are some issues that need to be solved with recursion into modules/packages. Getting reliable info on modules is harder than I thought. Also - decorating classes? Oh man.

I'm tackling all of the above now. Once I'm done, I - as a user - want to be able to

1. get a good idea of what's happening for debugging and coding
2. decorate everything with a general purpose timing decorator to see the actual performance of stuff in real code with statistics
3. be able to decorate everything fully annotated with numba.njit (and get an idea how much of a performance impact it has with 2.)
4. be able to decorate everything with beartype to track down typing issues that numba might complain about

[leycec]

Brython

This is the way.

As usual, feel free to steal the idea ;-)

If it's not nailed down, it's already in my backpack.

decorating classes? Oh man.

...heh.

So excited about all of this, @amogorkon! 2021 was a wild ride there, wasn't it? And 2022 is shaping up to be the growling tiger we really wish we weren't riding, but somehow here we are – arms flopping about uselessly and hoping this all ends well.

As for @leycec, I'm on the cusp of finally pushing out a pecuniary GitHub Sponsors profile. Then it's on to ReadTheDocs-hosted documentation that no longer embarrasses the entire @beartype organization. :sweat:

[amogorkon]

So excited about all of this, @amogorkon! 2021 was a wild ride there, wasn't it?

Indeed it was.

And 2022 is shaping up to be the growling tiger we really wish we weren't riding, but somehow here we are – arms flopping about uselessly and hoping this all ends well.

The writers of the series really outdid themselves this season. Hasn't been as nervewrecking in a long time..

As for @leycec, I'm on the cusp of finally pushing out a pecuniary GitHub Sponsors profile. Then it's on to ReadTheDocs-hosted documentation that no longer embarrasses the entire @beartype organization. 😓

Oh, nice! Wish you great success with that, you earned it 👍

Btw, I'm just pondering an idea regarding aspects. Web messages/exceptions really change the game in what's feasible in terms of usability. What if mod = use("some_mod" with mod @ beartype wouldn't actually apply anything but do a dry run ("what if..") and open a web message with different filters, options etc. for the user to tweak and then copy a proper generated use.apply_aspect(..) snippet to get exactly what they were looking for without guessing about what might or might not get hit?

[amogorkon]

I just went ahead and did it 😁

import use
from beartype import beartype
np = use("numpy")
np @ beartype

Now recurses through the module and tries to wrap (and unwrap) objects with the decorator, producing an html with the results like this: https://codepen.io/amogorkon/pen/QWaybPw

The options and filters for the generated snippet are still missing in the html, but I think at least the basic functionality is working now properly.

When I do

import use, numpy as np
use.apply_aspect(np, use.woody_logger)
a = np.zeros((2, 2))

it gives me

((2, 2),) {} -> numpy.zeros
(0.0,) {} -> numpy.isfinite
-> numpy.isfinite (in 13500 ns (1e-05 sec) -> True <class 'numpy.bool_'>
(0.0,) {} -> numpy.isfinite
-> numpy.isfinite (in 11300 ns (1e-05 sec) -> True <class 'numpy.bool_'>
(0.0,) {} -> numpy.isfinite
-> numpy.isfinite (in 9300 ns (1e-05 sec) -> True <class 'numpy.bool_'>
(0.0,) {} -> numpy.isfinite
-> numpy.isfinite (in 12400 ns (1e-05 sec) -> True <class 'numpy.bool_'>
-> numpy.zeros (in 17500 ns (2e-05 sec) -> [[0. 0.]
 [0. 0.]] <class 'numpy.ndarray'>

Which is pretty cool, isn't it 🎉

[leycec]

OMG. You are God-tier Pythonista. I'm not gonna lie, despite commonly doing so; this is phenomenal.

Ignoring @beartype integration for a hot minute, even your oh-so-punny use.woody_logger offers a countably infinite catalogue of insanely useful use cases. You've got embedded profiling, call stack tracing, parameter pretty printing, and so much more going on in that simple example. The killer app is that you didn't even have to instrument the target package itself; justuse just delicately reaches in and quietly instruments everything without the target package even knowing that there's a hand up its kilt.

Gotta ask, though we already know the answer: is any of this aspectization available in justuse 0.6.5 or are we all holding our breath for the next stable release to drop? Because I wanna start @beartype-ing everyone else's code without their consent.

Relatedly, I've opened up a prominent @beartype issue proposing that I document how exactly to go about this. Of course, we currently don't even have documentation in any meaningful sense. That should probably come first. :smiling_face_with_tear:

[amogorkon]

Hehehe. If you liked woody logger, behold the tinny profiler! I recall you asked for a way to measure the performance impact of applying beartype.

import use, numpy

use.apply_aspect(numpy, use.tinny_profiler)
numpy.zeros((3,3))
use.show_profiling()

produces https://codepen.io/amogorkon/pen/JjMbjgy - still needs a little polishing, but it's getting there. With that you should be able to assess the impact of applying decorators (I'm considering to add plots to show how timings change over time, but that's no priority atm).

This functionality isn't in 0.6.5, but I managed to get all our unit tests to pass today, so we're actually getting close to ready for the next release. We have a mass test (called "the beast") which attempts to use() pretty much all packages in the conda distribution (~2000 packages). I've simplified it so it all runs in a single parametrized test for 3-4 hours, installing everything in a single process. It's about 70% successful, which is okay-ish for now, but I'd like to have another look and see if there is any big issue we've missed so far that needs to be addressed before we can release it without breaking too much.

[amogorkon]

Relatedly, I've opened up a https://github.com/beartype/beartype/issues/114. Of course, we currently don't even have documentation in any meaningful sense. That should probably come first. 🥲

The way to apply decorators is going to be:

1. get an overview of what you're gonna hit with

   mod = use(package)
   mod @ decorator

   which will open a browser window with a list of all callables and you can adjust your filters, then

2. copy&paste the snippet back into your code, leading to a call like

   use.apply_aspect(module, decorator, **kwargs)

   with the added bonus that the aspectized module doesn't have to be use()d but can also be imported regularly.

[leycec]

We have a mass test (called "the beast") which attempts to use() pretty much all packages in the conda distribution (~2000 packages).

:fearful: → :open_mouth:

Your fearlessness is an example I strive to follow. Also, if I'm speed-reading this right...

use.apply_aspect(module, decorator, **kwargs)

...then @beartype users could conceivably configure the decoration by passing keyword arguments like so:

from beartype import (
    beartype, BeartypeConf, BeartypeStrategy)

# Forcefully stuff linear type-checking into "{module}",
# automating manual decorations like this:
#   @beartype(conf=BeartypeConf(strategy=BeartypeStrategy.On)
#   def muh_func(...): ...
use.apply_aspect(
    {module},
    beartype,
    conf=BeartypeConf(strategy=BeartypeStrategy.On),
)

Does that grok? Lastly, this fascinates my neglected inner child:

...with the added bonus that the aspectized module doesn't have to be use()d but can also be imported regularly.

Ah-ha! So:

• use.apply_aspect() dynamically monkey-patches in-place all decoratable attributes of the passed {module} by globally replacing all relevant source attributes of that module with their decorated target equivalents. Is that right? If so... that's sick, bro. So, good! Relatedly:
  • Is there a parallel API for side effect-free aspectization? Perhaps a similar use function that instead produces a proxy target module wrapper object without modifying the underlying source package or module? If so... wicked sick.
• Does that also transitively apply to all child submodules when {module} is in fact a package? If so... ultimate wicked sick!

[amogorkon]

from beartype import ( beartype, BeartypeConf, BeartypeStrategy)

use.apply_aspect( {module}, beartype, conf=BeartypeConf(strategy=BeartypeStrategy.On), )

@beartype(conf=BeartypeConf(strategy=BeartypeStrategy.On) My first thought was that it would simply translate to use.apply_aspect(module, beartype(conf=BeartypeConf(strategy=BeartypeStrategy.On)) With a nice and simple pre-initialized decorator, which is then applied to everything we can find.. but there's nothing simple about decorators, is it. Are you saying that your decorator has a side effect when initialized and that

decorator = beartype(conf=BeartypeConf(strategy=BeartypeStrategy.On)

def a(x:int) -> int:
   return x ** 2
def b(x:int) -> float:
   return x / 2

a = decorator(a)
b = decorator(b)

wouldn't work because it needs to be re-initialized? I hope not 🥺

use.apply_aspect() dynamically monkey-patches in-place all decoratable attributes of the passed {module} by globally replacing all relevant source attributes of that module with their decorated target equivalents. Is that right? If so... that's sick, bro.

@beartype
def _wrap(*, thing: Any, obj: Any, decorator: Callable, name: str) -> Any:
    wrapped = decorator(obj)
    _applied_decorators[(thing, name)].append(decorator)
    _aspectized_functions[(thing, name)].append(obj)

    # This will fail with TypeError on built-in/extension types.
    # We handle exceptions outside, let's not betray ourselves.
    setattr(thing, name, wrapped)
    return wrapped

That's all there is.. wait, are you trying to trick me into ast-fiddling?

{module} can be any object though, we just crawl all attributes looking for something of type(x) == type and recurse there. Getting the recursion right was tricky, so were names (qualnames vs. names vs no name at all.. 🚑 ) and a number of other nitty gritty details.. no wonder my newbies ran away 😭

Is there a parallel API for side effect-free aspectization? Perhaps a similar use function that instead produces a proxy target module wrapper object without modifying the underlying source package or module? If so... wicked sick.

What do you mean "side-effect free"? Ohhh.. you mean to decorate not replaceable attributes like numpy extension types? 🤔 Hmm.. use() already returns a ProxyModule, which proxies all method lookups transparently for other reasons already (working module reloading, for one - reminds me to look into aspectizing of ModuleReloader..), so we should be able to wire that in fairly easily. Although having said that, I'm having a feeling there might be a whole pit of dragons to deal with..

Does that also transitively apply to all child submodules when {module} is in fact a package? If so... ultimate wicked sick!

Eh. There is no distinction between package and module at this level. Whatever is imported is considered a module, but I guess you mean to decorate everything that is imported on the user's side.

It actually did and it could again if we removed

        try:
            thing_module_name = inspect.getmodule(thing).__name__
        except AttributeError:
            thing_module_name = "<unknown>"
        if thing_module_name != module_name:
            return

but it was a little too much raw power at that point. It would require more sophisticated Brython to process the thousands of objects in the DOM and display them in a non-headache-inducing manner. I guess the human side is the bottleneck for that one. Also for some reason applying a filter to the 2000 decoratable objects in numpy already takes a few seconds in the browser, which is a domain I yet have very little experience with.

[leycec]

use.apply_aspect(module, beartype(conf=BeartypeConf(BeartypeStrategy.On)))

This is truth. Your first thought is the correct thought. My first thought was derailed onto the rocks below by seeing **kwargs and then running amok with bad assumptions.

Since beartype(conf=BeartypeConf(...)) returns a decorator, the above use case looks great as is. :+1:

Are you saying that your decorator has a side effect when initialized...

Thank Freya, no. @beartype is idempotent (so, side-effect-free), because I'm still sane. I'm also pretty sure anyone publicly qualifying that they're "still sane" is one brick short of a full load.

wait, are you trying to trick me into ast-fiddling?

:eyes:

Actually, I think I just confused all of us.

Please do not disembowel justuse with AST fiddling. It's bad enough that beartype is eventually going to do that. This is a dark road we chosen few walk, @amogorkon.

setattr(thing, name, wrapped)

Ah-ha! So, use.apply_aspect() does indeed dynamically monkey-patch the target module in-place. I think. Actually, I'm still not certain, because my German is still trash despite having watched Run Lola Run 57 times. ^{possibly 58}

Just to be clear, does this snippet raise a BeartypeCallHintViolation exception?

import beartype, numpy, use
use.apply_aspect(numpy, beartype)
numpy.zeros('Pretty sure this is balls.')

Pretty sure it does, but I'm obsessively refreshing /r/Ukraine like a mad lad rather than just pulling and installing your master branch like I should be.

...you mean to decorate not replaceable attributes like numpy extension types?

I... actually no longer know what I mean. Not sure I ever did.

Nonetheless, the above question sounds like it leads to a risky and fragile place. This means I support it.

I guess you mean to decorate everything that is imported on the user's side.

Pretty sure I lost the plot a few comments ago. Note to self: never use the term "transitive" unless I myself know what that word means.

Basically, does use.apply_aspect() recurse into subpackages or does it just directly aspectize the passed module? As a selfish example that I now need to try, can we @beartype the entire beartype package (including all subpackages of that package) with a single one-liner like this:

import beartype, use
use.apply_aspect(beartype, beartype)

# Does the above aspectization recursively @beartype
# all functions defined throughout the entire "beartype"
# codebase? If so, the following should raise a
# "BeartypeCallHintViolation" exception.
beartype.vale.Is['This is full-on bollocks, bro.']

...or do we instead have to iteratively @beartype each specific beartype subpackage with another use.apply_aspect() call like this:

import beartype, use
use.apply_aspect(beartype, beartype)
use.apply_aspect(beartype.vale, beartype)  # <-- gotta do this?

# Please raise a "BeartypeCallHintViolation" exception.
beartype.vale.Is['This is full-on bollocks, bro.']

Unrelatedly, Panzerfaust. What a word, that word. Only German can succinctly express violent weaponry and its natural application. There's probably a historical lesson there.

[amogorkon]

import beartype, numpy, use use.apply_aspect(numpy, beartype) numpy.zeros('Pretty sure this is balls.')

Before you do this, you should do

from beartype import beartype
import use
np = use("numpy")
np @ beartype

which will attempt to apply the decorator (and unwrap the callable again) then open a browser tab with a gigantic list of everything callable and potentially decoratable, including info on exceptions that prevent decoration like

I could add the complete exception tracebacks to these, not sure if that would be very useful, though.

BUT use.apply_aspect doesn't raise exceptions itself because that make it very tedious to decorate a large module like numpy since you'd have to manually filter everything on that list that might cause problems.

Nonetheless, the above question sounds like it leads to a risky and fragile place. This means I support it.

Let me find a new minion to work on this.. >_>

Basically, does use.apply_aspect() recurse into subpackages or does it just directly aspectize the passed module? As a selfish example that I now need to try, can we @beartype the entire beartype package (including all subpackages of that package) with a single one-liner like this:

It doesn't recurse beyond the module border, at the moment. That means you'd have to do it like

import beartype, use
use.apply_aspect(beartype, beartype)
use.apply_aspect(beartype.vale, beartype)  # <-- gotta do this?

at least until I find a way to tame this raw power. It's definitely doable to recurse into everything, but it quickly becomes unpredictable because you'd be ehhh... wait a second.

Your usecase doesn't require recursion. The module name for beartype is beartype and the module name for beartype.vale is beartype.vale - and..

for mod in sys.modules: 
   if mod.split(".")[0] == "beartype": print(mod))

gives

beartype._decor
beartype.roar._roarexc
beartype.roar._roarwarn
beartype.roar
beartype._decor._code
beartype._util
beartype._util.text
beartype._util.text.utiltextmagic
beartype._decor._code.codesnip
beartype._decor._code._pep
beartype._decor._cache
beartype._util.cache
beartype._util.func
beartype._util.func.utilfuncwrap
beartype._util.utiltyping
beartype._util.func.utilfunccodeobj
beartype._util.func.utilfuncarg
beartype._util.utilobject
beartype._util.text.utiltextlabel
beartype._util.cache.utilcachecall
beartype._util.cls
beartype._util.cls.pep
beartype._util.cls.pep.utilpep3119
beartype._cave
beartype._cave._caveabc
beartype._util.py
beartype._util.py.utilpyversion
beartype._cave._cavefast
beartype._data
beartype._data.cls
beartype._data.cls.datacls
beartype._data.mod
beartype._data.mod.datamod
beartype._util.cls.utilclstest
beartype._util.mod
beartype._util.mod.utilmodimport
beartype._util.os
beartype._util.os.utilostest
beartype._util.py.utilpyinterpreter
beartype.meta
beartype._util.mod.utilmodtest
beartype._decor._cache.cachetype
beartype._util.cache.utilcacheerror
beartype._decor._code._pep._pepmagic
beartype._util.hint
beartype._util.hint.nonpep
beartype._util.hint.nonpep.utilnonpeptest
beartype._cave._cavemap
beartype._decor._code._pep._pepsnip
beartype._util.func.utilfuncscope
beartype._util.hint.pep
beartype._util.hint.pep.proposal
beartype._util.hint.pep.proposal.pep484585
beartype._util.hint.pep.proposal.pep484
beartype._util.hint.pep.proposal.pep484.utilpep484ref
beartype._util.mod.utilmodule
beartype._util.hint.pep.proposal.pep484585.utilpep484585ref
beartype._decor._code._pep._pepscope
beartype._util.cache.pool
beartype._util.cache.pool.utilcachepool
beartype._util.text.utiltextrepr
beartype._util.cache.pool.utilcachepoollistfixed
beartype._util.cache.pool.utilcachepoolobjecttyped
beartype._data.hint
beartype._data.hint.pep
beartype._data.hint.pep.sign
beartype._data.hint.pep.sign.datapepsigncls
beartype._data.hint.pep.sign.datapepsigns
beartype._data.hint.pep.sign.datapepsignset
beartype._util.hint.pep.proposal.pep484.utilpep484generic
beartype._util.hint.pep.proposal.pep484.utilpep484
beartype._util.hint.pep.proposal.utilpep585
beartype._util.hint.pep.proposal.pep484585.utilpep484585
beartype._util.hint.pep.proposal.pep484585.utilpep484585arg
beartype._util.hint.pep.proposal.pep484585.utilpep484585generic
beartype._util.hint.pep.proposal.pep484585.utilpep484585type
beartype._util.text.utiltextjoin
beartype._util.hint.pep.proposal.utilpep586
beartype.vale._factory
beartype._util.kind
beartype._util.kind.utilkinddict
beartype.vale._valevale
beartype.vale._factory._valeisabc
beartype.vale._factory._valeis
beartype.vale._util
beartype.vale._util._valeutilsnip
beartype.vale._factory._valeiscls
beartype.vale._factory._valeisobj
beartype.vale._factory._valeisoper
beartype.vale
beartype._util.hint.pep.proposal.utilpep593
beartype._data.hint.pep.datapeprepr
beartype._util.hint.pep.proposal.pep484.utilpep484newtype
beartype._util.hint.pep.utilpepget
beartype._util.hint.pep.utilpepattr
beartype._util.hint.pep.proposal.utilpep544
beartype._util.hint.pep.utilpeptest
beartype._data.func
beartype._data.func.datafunc
beartype._util.cache.map
beartype._util.cache.map.utilmapbig
beartype._util.hint.pep.proposal.pep484.utilpep484union
beartype._util.hint.utilhinttest
beartype._util.hint.utilhintconv
beartype._util.text.utiltextmunge
beartype._decor._code._pep._pephint
beartype._decor._error
beartype._decor._error._errorsleuth
beartype._decor._error._errortype
beartype._decor._error._pep
beartype._decor._error._pep._pep484
beartype._decor._error._pep._pep484._errornoreturn
beartype._decor._error._pep._pep484._errorunion
beartype._decor._error._pep._pep484585
beartype._decor._error._pep._pep484585._errorgeneric
beartype._decor._error._pep._pep484585._errorsequence
beartype._decor._error._pep._errorpep586
beartype._decor._error._pep._errorpep593
beartype._decor._error.errormain
beartype._util.func.utilfunctest
beartype._decor._data
beartype._decor._code._pep.pepcode
beartype._util.hint.pep.proposal.pep484585.utilpep484585func
beartype._decor._code.codemain
beartype._util.func.lib
beartype._util.func.pep
beartype._util.func.pep.utilpep484func
beartype._util.mod.lib
beartype._util.func.utilfuncstack
beartype._util.mod.lib.utilsphinx
beartype._util.func.lib.utilbeartypefunc
beartype._util.func.utilfuncmake
beartype._decor.main
beartype
beartype._util.text.utiltextident
beartype._decor._pep563
beartype._util.func.utilfuncfile
beartype._util.hint.pep.proposal.utilpep589

Which means we could simply pick stuff from sys.modules.. (and I wanted to get rid of the import caching, maybe finally a redeeming quality).

Unrelatedly, Panzerfaust. What a word, that word.

Fascinating how I was thinking the same thing the other day.. 🤯

[leycec]

BUT use.apply_aspect doesn't raise exceptions itself because that make it very tedious to decorate a large module like numpy since you'd have to manually filter everything on that list that might cause problems.

It is sane. Thus, this is the only way that matters.

That said, would an optional use.apply_aspect(..., *, warns: bool = False) boolean keyword parameter named warns (or something something) be trivial enough to make your precious time worthwhile? Passing use.apply_aspect(..., warns=True) might emit one non-fatal warning for each undecoratable object (possibly embedding the exception message originating that warning). The output would be gargantuan and stinky – which is why sane devs such as ourselves pipe that wretched mess into less.

Maybe? If easy-peasy, that could be nice because Pythonic and CLI-driven. If non-easy-peasy, we forget the last paragraph and return to binging Severance for the good of Apple revenue.

Your usecase doesn't require recursion.

Hold my beer:

>>> import beartype, sys
>>> for mod in sys.modules: 
...   if mod.split(".")[0] == "beartype": print(mod)
beartype.meta
beartype._decor
beartype._util
beartype._util.py
beartype._util.py.utilpyversion
beartype._util.cache
beartype.roar._roarexc
beartype.roar._roarwarn
beartype.roar
beartype._util.func
beartype._util.func.arg
beartype._util.func.utilfuncwrap
beartype._data
beartype._data.datatyping
beartype._util.func.utilfunccodeobj
beartype._util.func.arg.utilfuncargtest
beartype._util.text
beartype._util.utilobject
beartype._util.text.utiltextlabel
beartype._util.cache.utilcachecall
beartype.typing._typingpep544
beartype.typing
beartype._decor.conf
beartype._decor.cache
beartype._data.cls
beartype._cave
beartype._cave._caveabc
beartype._cave._cavefast
beartype._data.cls.datacls
beartype._decor._code
beartype._util.error
beartype._util.error.utilerror
beartype._decor._code.codemagic
beartype._util.cls
beartype._util.cls.pep
beartype._util.cls.pep.utilpep3119
beartype._data.mod
beartype._data.mod.datamod
beartype._util.cls.utilclstest
beartype._util.mod
beartype._util.mod.utilmodimport
beartype._util.mod.utilmodtest
beartype._decor.cache._cachetype
beartype._decor._error
beartype._util.hint
beartype._util.hint.nonpep
beartype._util.hint.nonpep.utilnonpeptest
beartype._cave._cavemap
beartype._data.hint
beartype._data.hint.pep
beartype._data.hint.pep.sign
beartype._data.hint.pep.sign.datapepsigncls
beartype._data.hint.pep.sign.datapepsigns
beartype._data.hint.pep.sign.datapepsignset
beartype._util.hint.pep
beartype._data.hint.pep.datapeprepr
beartype._util.hint.pep.proposal
beartype._util.hint.pep.proposal.pep484
beartype._util.hint.pep.proposal.pep484.utilpep484newtype
beartype._util.hint.pep.proposal.utilpep585
beartype._util.hint.pep.utilpepget
beartype._util.hint.pep.proposal.pep484.utilpep484
beartype._util.hint.pep.proposal.utilpep544
beartype._util.hint.pep.proposal.utilpep593
beartype._util.mod.utilmodule
beartype._util.hint.pep.utilpeptest
beartype._data.func
beartype._data.func.datafunc
beartype._util.cache.map
beartype._util.cache.map.utilmapbig
beartype._util.hint.pep.proposal.pep484.utilpep484union
beartype._util.hint.pep.proposal.utilpep557
beartype._util.hint.utilhinttest
beartype._util.hint.utilhintconv
beartype._decor._error._errorsleuth
beartype._util.text.utiltextmunge
beartype._util.text.utiltextrepr
beartype._util.hint.pep.proposal.pep484585
beartype._util.hint.pep.proposal.pep484.utilpep484ref
beartype._util.hint.pep.proposal.pep484585.utilpep484585ref
beartype._util.hint.pep.proposal.pep484585.utilpep484585arg
beartype._util.hint.pep.proposal.pep484585.utilpep484585type
beartype._util.text.utiltextjoin
beartype._decor._error._errortype
beartype._decor._error._pep
beartype._decor._error._pep._pep484
beartype._decor._error._pep._pep484._errornoreturn
beartype._decor._error._pep._pep484._errorunion
beartype._decor._error._pep._pep484585
beartype._util.hint.pep.proposal.pep484.utilpep484generic
beartype._util.hint.pep.proposal.pep484585.utilpep484585generic
beartype._decor._error._pep._pep484585._errorgeneric
beartype._util.hint.pep.proposal.pep484585.utilpep484585
beartype._decor._error._pep._pep484585._errorsequence
beartype._util.hint.pep.proposal.utilpep586
beartype._decor._error._pep._errorpep586
beartype._util.text.utiltextmagic
beartype._decor._error._pep._errorpep593
beartype._decor._error.errormain
beartype._util.func.utilfuncscope
beartype._util.func.utilfunctest
beartype._decor._call
beartype._decor._code.codesnip
beartype._decor._code._pep
beartype._decor._code._pep._pepmagic
beartype._util.func.arg.utilfuncargiter
beartype._decor._code._pep._pepsnip
beartype._decor._code._pep._pepscope
beartype._util.cache.pool
beartype._util.cache.pool.utilcachepool
beartype._util.cache.pool.utilcachepoollistfixed
beartype._util.cache.pool.utilcachepoolobjecttyped
beartype._util.kind
beartype._util.kind.utilkinddict
beartype._decor._code._pep._pephint
beartype._util.hint.pep.proposal.pep484585.utilpep484585func
beartype._decor._code.codemain
beartype._util.cls.pep.utilpep557
beartype._util.func.lib
beartype._util.func.pep
beartype._util.func.pep.utilpep484func
beartype._util.mod.lib
beartype._util.func.utilfuncstack
beartype._util.mod.lib.utilsphinx
beartype._util.func.lib.utilbeartypefunc
beartype._util.func.utilfuncmake
beartype._decor._core
beartype._decor.cache.cachedecor
beartype._decor.main
beartype

Thankfully (or not), beartype.vale isn't there. Praise Jeebus! B-b-but... how could this be? Well, it turns out that the beartype codebase goes out of its way to import as little as possible at global scope.

^{Obviously, we failed. I mean, just look at that list. We're still internally importing way too much at global scope. So that I can sleep tonight, let's pretend I didn't see that.}

This includes beartype.vale, which is (mostly) only ever imported when a user explicitly imports from that subpackage.

Annnnnyway. Recursion is an understandable nest of vipers in Python, because we don't have tail recursion or other obvious (but necessary) optimizations. You can't safely even do recursion in Python without blowing up the stack.

This is why I'm now going to drown my sorrows in a bucket of Netflix-fueled anime. Time to forget my troubles, everybody! :tv:

[amogorkon]

That said, would an optional use.apply_aspect(..., *, warns: bool = False) boolean keyword parameter named warns (or something something) be trivial enough to make your precious time worthwhile? Passing use.apply_aspect(..., warns=True) might emit one non-fatal warning for each undecoratable object (possibly embedding the exception message originating that warning). The output would be gargantuan and stinky – which is why sane devs such as ourselves pipe that wretched mess into less.

Sounds good. Question is whether you'd prefer raise or warn or print or a combination of those. If you're thinking of piping it anyway, I could do you one better and add an output or file keyword argument which could be plugged into print() to pipe the output to the specified target directly. Then you'd get a

use.apply_aspect(mod, beartype, warns=True, output="bear-problems.log")

Annnnnyway. Recursion is an understandable nest of vipers in Python, because we don't have tail recursion or other obvious (but necessary) optimizations. You can't safely even do recursion in Python without blowing up the stack.

Well, technically this is true for deep recursion, but we're not even talking about very deep recursion here so it is an option actually. .oO(Maybe you're crazy enough to come up with a viable solution 😈 ) The issue with recursion in this case is this: When you're starting to check an object like a module for a __dict__ to recurse into, you're first recursing into all the imports (stdlib, external libs, everything) but also into class-parents, including object itself and from there into everything - everything is connected with each other and before you know it you're decorating the whole planet 💥 . So, I tried to come up with ways to limit the effects in some meaningful way, but it's kinda hard to balance between too strict and too lax. I definitely see the benefits of being less strict, but 1) how to communicate this to the user (displaying and filtering 10k objects in a browser may not be easily feasible) 2) any ideas what could be meaningful limits?

I'm thinking that we could keep the module (most strict) as default "blast radius" but have a keyword with an enum flag to increase the affected area like use.apply_aspect(mod, beartype, affect=package | dependencies) or somesuch.. What do you think?

[leycec]

Ugh. Just realized I never replied to this awesomeness. Since I'm exhausted, that can't happen tonight. Let's rain snow check for this a deep dive later this week. :thinking:

Thanks as always for being so inspirational, irrepressible, and smart, @amogorkon.

[leycec]

Phew! What a week, that last week. Let's discombobulate this.

Question is whether you'd prefer raise or warn or print or a combination of those.

I choose you, warn and/or warns! That said, that's @beartype for you. We raise exceptions all over the place. raise would destroy everything. For other decorators that raise exceptions less frequently (...so, less fragile decorators is what I'm saying), raise could very well be the Right Thing.™

print is probably subsumed by either raise or warn – maybe? After all, warnings and exceptions can always be redirected to the requisite open file handle. So, raise and warn >>>>> print. ^maybe

I could do you one better and add an output or file keyword argument which could be plugged into print() to pipe the output to the specified target directly.

Yes. I vote for file, but a file parameter whose value must be an open file handle rather than a stringified filename. This approach is the Big Boss, because users can then pass anything that satisfies the typing.TextIO protocol – including io.StringIO objects and the standard sys.stdout and sys.stderr streams: e.g.,

from sys import stderr

# Punt all decoration exceptions to stderr, yo!
use.apply_aspect(mod, beartype, file=stderr)

# Punt all decoration exceptions to my 1TB-large logfile!
with open('bear-problems.log') as you_didnt_see_nuffin:
    use.apply_aspect(mod, beartype, file=you_didnt_see_nuffin)

...before you know it you're decorating the whole planet :boom: .

:exploding_head:

I'm thinking that we could keep the module (most strict) as default "blast radius"...

Yup. Explicit is better than implicit. Minimizing the explosive potential minimizes user migraines and old-age tinnitus. Thanks a lot, decades of reckless listening to German power metal at unhealthy volumes!

Relevantly, I just tripped across this deep StackOverflow post non-recursively iterating over the full transitive set of all submodules of a given package under Python ≥ 3.6:

import importlib, pkgutil, sys

def find_abs_modules(module):
    path_list = []
    spec_list = []
    for importer, modname, ispkg in pkgutil.walk_packages(module.__path__):
        import_path = f"{module.__name__}.{modname}"
        if ispkg:
            spec = pkgutil._get_spec(importer, modname)
            importlib._bootstrap._load(spec)
            spec_list.append(spec)
        else:
            path_list.append(import_path)
    for spec in spec_list:
        del sys.modules[spec.name]
    return path_list

Of course, there's a bit of unhealthy modification of global state (e.g., sys.modules) up there – but anyone deploying justuse-based aspectization in the first place is probably already prepared for bombshells like that.

...have a keyword with an enum flag to increase the affected area like use.apply_aspect(mod, beartype, affect=package | dependencies)

Totally fascinating. Love that |-style union set operator abuse, too. Alternately, you could also defer responsibility to the user with a similar parameter accepting an explicit blast radius of all package and module names to be aspectized: e.g.,

import use
use.apply_aspect(mod, beartype, affect=use.iter_submodules(mod))

Here, the hypothetical use.iter_submodules() function (which probably just resembles the above find_abs_modules() function) either yields or returns an iterable of the fully-qualified names of all submodules and subpackages of the passed module or package.

Ultimate API victory is now within sight! :hushed:

[amogorkon]

Wow, lots of good stuff right there. Hmm.. your hypothetical iter_submodules function intrigues me. What if apply_aspect could take an iterable[mod] as first parameter instead? We could simplify the signature to apply_aspect(mod: Any | Iterable[ModuleType], decorator: Callable) special casing Iterable[ModuleType] so you could (actually, using the brilliant https://github.com/robinhilliard/pipes) do use.iter_submodules(mod) >> use.apply_aspect(beartype) (or more traditionally use.apply_aspect(use.iter_submodules(mod), beartype))

[leycec]

What if apply_aspect could take an iterable[mod] as first parameter instead?

So much, "YESSSS!!!" Generalize that first parameter up.

https://github.com/robinhilliard/pipes

...what is this madness I've always yearned for in my life?

Where did you even find out about that hidden gem, if I might pry deeply into your clandestine browsing habits? pipes only has 48 GitHub stars, so it's still early days for them. Incredibly impressive regardless; the first thing I thought when I saw that deadly syntactic brew wasn't Elixir-style piping but C++-style output stream redirection. The nostalgia is profound.

Actually, I'm wondering now if the latter inspired the former... Of course, no one would (or should) ever publicly admit that. ^...heh

use.iter_submodules(mod) >> use.apply_aspect(beartype)

:hot_face:

use.apply_aspect(use.iter_submodules(mod), beartype)

:sunglasses:

If you do end up leveraging the @pipes.pipes decorator, consider only doing so conditionally when pipes is installed as an optional runtime dependency. I've learned through the ongoing school of real-world pain to avoid mandatory dependencies wherever feasible – which is everywhere.

So excited about all this awesome sauce, @amogorkon.

[amogorkon]

Where did you even find out about that hidden gem, if I might pry deeply into your clandestine browsing habits? pipes only has 48 GitHub stars, so it's still early days for them.

I was venturing into functional-land and was looking for a way to untangle a deeply nested (()) mess. The obvious (but imo wrong) solution is to overload | to wrap __call__ etc. which I've also initially fiddled with, and there's also https://pypi.org/project/pipe/ which does this, but the problem with this approach is that many built-in functions (like map) have their second/last positional argument to iterate over, which requires rewriting a lot of code (or monkeypatching). Another issue with this approach is that it doesn't really make the code simpler in my opinion but introduces even more convolution. look at print(sum(range(100) | select(lambda x: x ** 2) | where(lambda x: x < 100))) (from pipe README) and compare that to

@pipes
def foo():
    (
    range(100)
    << map(lambda x: x**2)
    << filter(lambda x: x < 100)
    >> sum
    >> print
    )

so, yeah. The << and >> need a bit of getting used to, but it's a small sacrifice compared to rewriting those builtins, I think, and it's much cleaner. Ever since I found it, I've been lobbying for it, but there's clearly a visibility problem (probably also because it was labelled beta and it hasn't been touched in 4 years, but considering it's just 50 lines of code and it just works...)

If you do end up leveraging the @pipes.pipes decorator, consider only doing so conditionally when pipes is installed as an optional runtime dependency. I've learned through the ongoing school of real-world pain to avoid mandatory dependencies wherever feasible – which is everywhere.

Oh no, I wouldn't force a dependency like this on anyone! However, pipes is so small, I actually pasted the few lines in https://github.com/amogorkon/justuse/blob/unstable/src/use/tools.py, putting it to good use in https://github.com/amogorkon/justuse/blob/unstable/src/use/pimp.py 😁 👼

[amogorkon]

I'm just toying with all the cool new ideas we came up with, but your suggested iter_submodules turns out to be much trickier than I thought, since numpy seems to be allergic to any importlib._bootstrap._load(spec) tricks (it simply crashes the program without any exception whatsoever). If I remove the module import-sideeffect like

def iter_submodules(module: ModuleType) -> Iterable[ModuleType]:
    for importer, modname, ispkg in pkgutil.walk_packages(module.__path__):
        if not ispkg and (mod := sys.modules.get(f"{module.__name__}.{modname}")):
            yield mod

this is OK for numpy, but it only produces a minimal list of modules for beartype (meta)...

[amogorkon]

I think I finally figured it out. This is much more involved than I anticipated, but I think this is actually what we want..

import ast, contextlib, sys
from pathlib import Path
import matplotlib

def is_builtin(name, mod):
    if name in sys.builtin_module_names:
        return True

    if hasattr(mod, "__file__"):
        relpath = Path(mod.__file__).parent.relative_to(
                (Path(sys.executable).parent / "lib"))
        if relpath == Path():
            return True
        if relpath.parts[0] == "site-packages":
            return False
    return True

def get_imports_from_module(mod):
    if not hasattr(mod, "__file__"):
        return
    with open(mod.__file__, "rb") as file:
        with contextlib.suppress(ValueError):
            for x in ast.walk(ast.parse(file.read())):
                if isinstance(x, ast.Import):
                    name = x.names[0].name
                    if (mod := sys.modules.get(name)) and not is_builtin(name, mod):
                        yield name
                if isinstance(x, ast.ImportFrom):
                    name = x.module
                    if (mod := sys.modules.get(name)) and not is_builtin(name, mod):
                        yield name

def get_submodules(mod, visited=None, results=None):
    if results is None:
        results = set()
    if visited is None:
        visited = set()
    for name in get_imports_from_module(mod):
        if name in visited:
            continue
        visited.add(name)
        results.add(name)
        for x in get_imports_from_module(sys.modules[name]):
            results.add(x)
            get_submodules(sys.modules[x], visited, results)
    return results

for get_submodules(matplotlib) this actually returns

{'numpy.__config__', 'matplotlib', 'cycler', 'PIL', 'numpy.core._multiarray_umath', 'matplotlib.fontconfig_pattern', 'matplotlib.cbook', 'matplotlib._api', 'PIL.PngImagePlugin', 'pyparsing', 'matplotlib._api.deprecation', 'numpy', 'packaging.version', 'matplotlib.rcsetup', 'matplotlib._cm', 'matplotlib._cm_listed', 'matplotlib.colors', 'matplotlib.cm', 'matplotlib._enums', 'numpy._pytesttester'}

which means it really seems to hit all non-builtin dependencies that are currently imported without any side-effects (disregarding the recursion-function >_>) and no weird hidden imports.

for get_submodules(beartype) this list is a little longer:

{'beartype._cave._caveabc', 'beartype._cave._cavefast', 'beartype._cave._cavemap', 'beartype._data.cls.datacls', 'beartype._data.func.datafunc', 'beartype._data.hint.pep.datapeprepr', 'beartype._data.hint.pep.sign', 'beartype._data.hint.pep.sign.datapepsigncls', 'beartype._data.hint.pep.sign.datapepsigns', 'beartype._data.hint.pep.sign.datapepsignset', 'beartype._data.mod.datamod', 'beartype._decor._cache.cachetype', 'beartype._decor._code._pep._pephint', 'beartype._decor._code._pep._pepmagic', 'beartype._decor._code._pep._pepscope', 'beartype._decor._code._pep._pepsnip', 'beartype._decor._code._pep.pepcode', 'beartype._decor._code.codemain', 'beartype._decor._code.codesnip', 'beartype._decor._data', 'beartype._decor._error._errorsleuth', 'beartype._decor._error._errortype', 'beartype._decor._error._pep._errorpep586', 'beartype._decor._error._pep._errorpep593', 'beartype._decor._error._pep._pep484._errornoreturn', 'beartype._decor._error._pep._pep484._errorunion', 'beartype._decor._error._pep._pep484585._errorgeneric', 'beartype._decor._error._pep._pep484585._errorsequence', 'beartype._decor._error.errormain', 'beartype._decor.main', 'beartype._util.cache.map.utilmapbig', 'beartype._util.cache.pool.utilcachepool', 'beartype._util.cache.pool.utilcachepoollistfixed', 'beartype._util.cache.pool.utilcachepoolobjecttyped', 'beartype._util.cache.utilcachecall', 'beartype._util.cache.utilcacheerror', 'beartype._util.cls.pep.utilpep3119', 'beartype._util.cls.utilclstest', 'beartype._util.func.lib.utilbeartypefunc', 'beartype._util.func.pep.utilpep484func', 'beartype._util.func.utilfuncarg', 'beartype._util.func.utilfunccodeobj', 'beartype._util.func.utilfuncmake', 'beartype._util.func.utilfuncscope', 'beartype._util.func.utilfuncstack', 'beartype._util.func.utilfunctest', 'beartype._util.func.utilfuncwrap', 'beartype._util.hint.nonpep.utilnonpeptest', 'beartype._util.hint.pep.proposal.pep484.utilpep484', 'beartype._util.hint.pep.proposal.pep484.utilpep484generic', 'beartype._util.hint.pep.proposal.pep484.utilpep484newtype', 'beartype._util.hint.pep.proposal.pep484.utilpep484ref', 'beartype._util.hint.pep.proposal.pep484.utilpep484union', 'beartype._util.hint.pep.proposal.pep484585.utilpep484585', 'beartype._util.hint.pep.proposal.pep484585.utilpep484585arg', 'beartype._util.hint.pep.proposal.pep484585.utilpep484585func', 'beartype._util.hint.pep.proposal.pep484585.utilpep484585generic', 'beartype._util.hint.pep.proposal.pep484585.utilpep484585ref', 'beartype._util.hint.pep.proposal.pep484585.utilpep484585type', 'beartype._util.hint.pep.proposal.utilpep544', 'beartype._util.hint.pep.proposal.utilpep585', 'beartype._util.hint.pep.proposal.utilpep586', 'beartype._util.hint.pep.proposal.utilpep593', 'beartype._util.hint.pep.utilpepattr', 'beartype._util.hint.pep.utilpepget', 'beartype._util.hint.pep.utilpeptest', 'beartype._util.hint.utilhintconv', 'beartype._util.hint.utilhinttest', 'beartype._util.kind.utilkinddict', 'beartype._util.mod.lib.utilsphinx', 'beartype._util.mod.utilmodimport', 'beartype._util.mod.utilmodtest', 'beartype._util.mod.utilmodule', 'beartype._util.os.utilostest', 'beartype._util.py.utilpyinterpreter', 'beartype._util.py.utilpyversion', 'beartype._util.text.utiltextjoin', 'beartype._util.text.utiltextlabel', 'beartype._util.text.utiltextmagic', 'beartype._util.text.utiltextmunge', 'beartype._util.text.utiltextrepr', 'beartype._util.utilobject', 'beartype._util.utiltyping', 'beartype.meta', 'beartype.roar', 'beartype.roar._roarexc', 'beartype.roar._roarwarn', 'beartype.vale._valevale'}

is this the list of modules you were looking for?

[amogorkon]

Aaaaaand another breakthrough! 😄 Behold - working class decorators!

def woody_logger(thing: Callable) -> Callable:
    """
    Decorator to log/track/debug calls and results.

    Args:
        func (function): The function to decorate.
    Returns:
        function: The decorated callable.
    """
    if isinstance(thing, type):
        class wrapper(thing.__class__):
            def __new__(cls, *args, **kwargs):
                print(f"{args} {kwargs} -> {thing.__name__}()")
                before = perf_counter_ns()
                res = thing(*args, **kwargs)
                after = perf_counter_ns()
                print(
                    f"-> {thing.__name__}() (in {after - before} ns ({round((after - before) / 10**9, 5)} sec) -> {type(res)}"
                )
                return res

    else:
        @wraps(thing)
        def wrapper(*args, **kwargs):
            print(f"{args} {kwargs} -> {_qualname(thing)}")
            before = perf_counter_ns()
            res = thing(*args, **kwargs)
            after = perf_counter_ns()
            print(
                f"-> {_qualname(thing)} (in {after - before} ns ({round((after - before) / 10**9, 5)} sec) -> {res} {type(res)}"
            )
            return res

    return wrapper

[leycec]

So much goodness to unpack here.

I was venturing into functional-land...

Ahh, yes. The age-old software idiom for "In a land far, far away..." is how all good stories start.

Lemme grab the stale week-old popcorn. :popcorn:

...and was looking for a way to untangle a deeply nested (()) mess.

You're triggering my Emacs Lisp PTSD here.

True story: Emacs pinky physically crippled the pinky fingers on both of my hands... permanently. Don't repeat my mistakes, younger generation. Friends don't let friends Emacs.

However, pipes is so small, I actually pasted the few lines in...

This is the way. The @beartype codebase is littered with the bones of other dead projects I've picked clean of juicy marrow and assimilated into my own skeletal frame.

:meat_on_bone: :arrow_right: :bone:

The << and >> need a bit of getting used to...

No, you've nailed it. That's really a brilliant take: since a single overloaded symbol like | fails to generalize to divergent calling conventions (e.g., passing an operand as the second rather than first parameter to a function), only a pair of symmetric overloaded symbols like >> and << suffices.

is this the list of modules you were looking for?

Hole-in-one. That's exactly it. I applaud thunderously from the bleachers while also acknowledging the immense buckets of sweat, blood, and tears that probably went into making that AST import parser happen. Your genius is on a raised plateau of its own.

The upcoming beartype 0.11.0 release will probably also implement some form of recursive import hooking to magically apply @beartype to an entire package or module. I must warn you: your code is hot, I trust you more than I do me, and the <Ctrl>, <c>, and <v> keys on my keyboard are worn out from decades of furious copy-pasting.

What I'm saying here is... I may desperately pilfer the AST import parser you hand-built above for our own perfidious purpose. If I do this, I will loudly point everyone towards justuse and advise they use your package instead of our crude facsimile. :loudspeaker: :wavy_dash:

Lastly, this is...

def woody_logger(thing: Callable) -> Callable:
    ...
    if isinstance(thing, type):
        class wrapper(thing.__class__):
            def __new__(cls, *args, **kwargs):
                ...

Forehead-smacking cleverness! You're now going to hate me just a little bit harder, but... it's apparently possible for thing.__class__ to prohibit subclassing through metaclass trickery: e.g.,

>>> class FinalMeta(type):
...    def __new__(cls, name, bases, classdict):
...        for b in bases:
...            if isinstance(b, Final):
...                raise TypeError("type '{0}' is not an acceptable base type".format(b.__name__))
...        return type.__new__(cls, name, bases, dict(classdict))
>>> class FinalType(metaclass=FinalMeta): ...
>>> class ThisRaisesAHorribleException(FinalType): ...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __new__
TypeError: type 'FinalType' is not an acceptable base type

It's best to forget you saw that. I already did, in fact.

[amogorkon]

Go ahead and take the code you need 👍 Can't hurt to get more exposure and people testing things. Even though you wouldn't need justuse anymore to aspectize stuff with beartype, it wouldn't mean the end for me to work on aspectizing though - beside the woody_logger, I've got the tinny_profiler in the pipe for interactive profiling of code and I'm thinking about a "application health" thing that tracks calls and applies wavelet analysis to give an overview of what's happening. I'd be lying though if I said I wouldn't miss working with you on this, it's been a pleasure and honor, milady 😺

Thanks for the heads-up on the FinalMeta thing.. and the headache😛