Using inspect module to detect signature and reorganize the function arguments

[Ricyteach]

Hi Graham, I just discovered this project and I love it. Would you be willing to comment on how to properly do something?

Here is a non-wrapt decorator to be applied to a function stub:

import inspect
from functools import wraps

def my_decorator(func):
    sig = inspect.signature(func)

    @wraps(func)
    def wrapped(*args, **kwargs):
        # re-order args and kwargs into same order as function signature
        bound = sig.bind(*args, **kwargs)
        # do calculation using correct order
        return do_the_calc(*bound.arguments.values())

    return wrapped

I apply it like this:

@my_decorator
def func(a, b, c):
    """I am a stub; look for implementation elsewhere."""
    ...

This is a stub because I am only using it to customize the function signature. The real implementation is hidden in the do_the_calc function (which is really a dynamically created function that accepts different numbers of positional-only args in the signature), for reasons I won't go into.

However since this non-wrapt decorator isn't a descriptor, it has a bug when I try to use it in a class body:

class C:
    @my_decorator
    def method(self, a, b, c):
        """I am a stub; look for implementation elsewhere."""
        ...

...self gets added to the args.

The reason it is in a class body is because the class will have other functionality related to the method. Also note that it is true the method could probably be a staticmethod, but I'd rather not require that in the API I am writing, so I am using a regular method for my test suite.

Obviously the bug is fixed when I use the wrapt project (BRAVO!) -- self is removed from the args:

import inspect
import wrapt

@wrapt.decorator
def my_decorator(func, instance, args, kwargs):
    sig = inspect.signature(func)
    # re-order args and kwargs into same order as function signature
    bound = sig.bind(*args, **kwargs)
    # do calculation using correct order
    return do_the_calc(*bound.arguments.values())

class C:
    @my_decorator
    def method(self, a, b, c):
        """I am a stub; look for implementation elsewhere."""
        ...

Here is the only problem I see with this:

The signature function is being run for every function call now. I would prefer it just run once-- when the method/function is initially wrapped. This probably isn't possible since wrapt doesn't know it's a method at the time it is called, but what I am wondering is, is there perhaps another feature in wrapt that addresses this situation? Or do you have any other suggestion on how to properly do this so I am not having to call signature() at every function call?

[Ricyteach]

By the way: I have been reading the docs here which seems to be related to what I am trying to do:

https://wrapt.readthedocs.io/en/latest/decorators.html#signature-changing-decorators

...but my head is spinning on how to adapt it to my particular situation. I think I can get it eventually if I stare at it long enough but help would be greatly appreciated. :)

EDIT: I think I am getting close. The below works, but is there a better way? It would be nice to get rid of that ismethod() call since it theoretically should only be needed ONCE for this decorated function.

import inspect
import wrapt

def my_decorator(func):

    sig = inspect.signature(func)

    @wrapt.decorator
    def wrapper(wrapped, instance, args, kwargs):
        is_a_method = inspect.ismethod(wrapped)
        # re-order args and kwargs into same order as function signature
        bound = sig.bind(instance, *args, **kwargs) if is_a_method else sig.bind(*args, **kwargs)
        # do calculation using correct order
        return do_the_calc(*list(bound.arguments.values())[is_a_method:])

    return wrapper(func)

[GrahamDumpleton]

I'll try and read all this properly later when have time, but you can skip the ismethod() call as you can work it out from what the instance argument is set to. See:

• https://wrapt.readthedocs.io/en/latest/decorators.html#universal-decorators

It worries me a bit that such a check is needed in your case though and makes me think you are going about this all wrong since wrapt works on premise that you get a bound function already so you don't have to worry about whether it is a method or not, at least not to work out what to do with any self argument. Knowing the context may be useful in other circumstances though, but not usually related to binding.

Do you have a concrete standalone example in one file that runs which you can supply so can look at it better.

[GrahamDumpleton]

Also see:

• https://wrapt.readthedocs.io/en/latest/decorators.html#processing-function-arguments

Using inspect module to bind arguments will be more expensive. You are better off using mechanisms described in that section. Specifically the two examples of:

@wrapt.decorator
def my_decorator(wrapped, instance, args, kwargs):
    def _execute(arg1, arg2, *_args, **_kwargs):

        # Do something with arg1 and arg2 and then pass the
        # modified values to the wrapped function. Use 'args'
        # and 'kwargs' on the nested function to mop up any
        # unexpected or non required arguments so they can
        # still be passed through to the wrapped function.

        return wrapped(arg1, arg2, *_args, **_kwargs)

    return _execute(*args, **kwargs)

and:

@wrapt.decorator
def my_decorator(wrapped, instance, args, kwargs):
    def _arguments(arg1, arg2, *args, **kwargs):
        return (arg1, arg2)

    arg1, arg2 = _arguments(*args, **kwargs)

    # Do something with arg1 and arg2 but still pass through
    # the original arguments to the wrapped function.

    return wrapped(*args, **kwargs)

In other words, use a nested function call to force binding of keyword arguments to named arguments etc.

[Ricyteach]

Thank you for your time, and thank you for your suggestions.

10,000 ft BASIC SUMMARY OF WHAT I BELIEVE I WANT TO DO: in the decorator body, I need to use the signature of function A-- which is the wrapped function-- to reorder the args and kwargs, then feed them positionally into function B for evaluation. Function B has no idea what the signature of A looks like-- it just expects the arguments in a certain positional order. Function A has no idea what the calculation is to be; it is just there to provide a signature with a nice interface for the user, so they don't have to deal with function B.

I am focusing the rest of this post on providing the detailed info you asked for.

Here is a gist showing what I am trying to do (requires numpy and of course, wrapt):

https://gist.github.com/Ricyteach/41bd75d51c96f8ed9cd8b76d8b7da65c

Longer summary (using the wrapt functionality for step 3):

1. It is an API for conveniently expressing the concept of a load combination (field of civll engineering). Examples of some factored load combinations (ie, "factored" means the loads have been increased to account for statistical variance): a. dead load only: 1.4D b. dead load and live load and a companion load: 1.2D + 1.6L + 0.5(Lr or S) c. dead load and roof load: 1.2D + 1.6(Lr or S) + (L or 0.5W) d. dead load and wind load: 1.2D + 1.0W + L + 0.5(Lr or S)

2. To express a combination we can compose it like this using bitwise operators (this is example b. from step 1 above):

   >>> D, L, Lr, S = (Factored(s) for s in "D L Lr S".split())
   >>> DL_and_LL_expr = 1.2*D & 1.6*L & 0.5*(Lr | S)

   And calculate an actual load combination like this, where the load combination matrix on the LHS is @ multiplied by a 1-D matrix representing the values of the loads in the RHS:

   >>> DL_and_LL_expr.matrix @ [1, 2, 3, 4]
   array([5.9, 6.4])

   In this example, D is 1, L is 2, Lr is 3, and S is 4.

3. Use the load combination decorator to assign the above (step 2) matrix multiplication operation to a function with a nice signature.

   >>> @load_combination(1.2*D & 1.6*L & 0.5*(Lr | S))
   ... def primary_live_load(D, L, Lr, S):
   ...     pass
   ... 
   >>> primary_live_load(1, 2, 3, 4)
   array([5.9, 6.4])

The body of the load_combination wrapper executes the matrix multiplication step (2). The only thing needed from primary_live_load is the function meta data (signature, docstring, all that stuff).

[Ricyteach]

Thank you for the two examples you gave. They seem like they are on the right track but after staring at them for a while I think the only way I can use them to solve this problem is by changing the API that I would like to have. I would have to change things so the user has to write code like this:

@load_combination(1.2*D & 1.6*L)
def combo(D, L):
    return D, L

...rather than writing it like this:

@load_combination(1.2*D & 1.6*L)
def combo(D, L):
    ...

The second version is greatly preferred.

SOME MORE DISCUSSION BELOW: MIGHT BE TL/DR, if so then sorry.

What we have to do is a matrix multiplication step (see step 2 in post above; it is this line in the gist; this is the point at which we have to have the arguments in the right order to supply them as a right-hand-side 1-D matrix for multiplication):

#   left-hand-side @ right-hand-side
combination_matrix @ loads

So the basic issue seems to be that the matrix multiplication action requires that the loads object has to be a set of positional/sequential arguments supplied in the right order (ie, the order of the columns of the combination_matrix representing the load combination).

But at runtime, the decorated function can be called using a mixture of positional and keyword arguments in a different order:

@load_combination(D & L & (Lr | S)
def my_combo(D, L, Lr, S): ...

# function called in the wrong order
combo_result = my_combo(Lr=3, S=4, L=2, D=1)

So we have to first bind the supplied arguments to the function signature in order to get them in the right order, THEN use them in the correct order to execute the matrix multiplication step. This next snippet performs same action as the my_combo function call above:

# COLUMN LABELS ON NEXT LINE:
#                D  L Lr  S
mat = np.array([[1, 1, 1, 0],
                [1, 1, 0, 1]])
# COLUMN LABELS ON NEXT LINE:
#                   D, L,Lr,S
mat_result = mat @ [1, 2, 3, 4]
assert combo_result == mat_result

Note that order has to be corrected, from:

my_combo(Lr=3, S=4, L=2, D=1)

...to:

my_combo(1, 2, 3, 4)

...and this correction has to happen at runtime.

[GrahamDumpleton]

Yeah, looked at it and you are doing some really weird stuff. See if you can at least eliminate is_a_method check by consulting instance instead. So use:

        if instance and not inspect.isclass(instance):
            ... this is an instance method

[Ricyteach]

I'll definitely eliminate that, agree with you it isn't needed. Thanks!

[GrahamDumpleton]

Are you all good with this and work out an adequate solution?