How to properly define a function with decorator

[DogLooksGood]

Hi, I'm trying to convert a Python code which define a function with decorator.

@d
def f(v=x):
   return v ** 2

I have two questions about this.

How to define a function with decorator, I came up with something like

(def f
  (d (py/make-tuple-function (fn [x] (np/power x 2)))))

But I got an error that what py/make-tuple-function returns is an unsupported callable.

Second, what ** in this python code, is overloaded as numpy's power I think. So is this approach suitable?

Thanks for all the hard work in this awesome library.

[jjtolton]

To perform **:

(require '[libpython-clj.require :refer [require-python]]_
(require-python '[operator :as op])
(op/pow 2 6)
;;=> 64

On the decorator, I'm not sure I understand. Here are some possibilities:

(let [{{:strs [add1]} :globals} (py/run-simple-string
                                 "
def add1(f):
    def _add1(*args, **kwargs):
        res = f(*args, **kwargs)
        return res + 1
    return _add1

")]
  (def add1 add1))

(def add2ab (add1 (fn [a b] (+ a b))))

(add2ab 1 2)
;; => 4

But a Python decorator is simply a function that takes an argument as a function and returns an argument as a result, which can be done in Clojure without the need for Python. You can freely mix Python decorators with Clojure functions.

(let [{{:strs [times3]} :globals} (py/run-simple-string
                                 "
def times3(x):
    return x * x * x

")]
  (def cubed times3))

(defn div2 [f]
  (fn [& args]
    (/ (apply f args)
       2)))

(def cubed-div2 (div2 cubed))

(cubed-div2 2)
;;=> 4

Is that what you were looking for?

[DogLooksGood]

I still have a problem here. This is the real code I need to write.

@pymc.deterministic
def some_variable(p=x):
    return 0.5 * p + 0.25

This pymc.deterministic is something provided by a library, and the default argument value x should be available, in my case, x is defined in Clojure side with API in Python.

If I use py/run-simple-string, I have to add some import statement, and this x is not available in Python side.

---

I also found the function created by py/->py-fn or py/make-tuple-fn, seems lacking the information for signature when check with inspect._signature_from_callable.

[jjtolton]

I might need some more information.

For instance, what is pymc? Do you have access to it from the Clojure code?

(def my-x 42)
(require-python 'pymc)
(def some-variable 
    (pymc/deterministic 
        (fn some-variable ([] (some-variable my-x)) ([p] (* 0.5 (+ p 0.25))))))

[cnuernber]

We do not put the same metadata on Python functions as what happens if you define a function in Python. I think we could put more metadata on functions for sure.

[DogLooksGood]

pymc is a library for bayesian stuff. and this is its repo:

https://github.com/pymc-devs/pymc3

And yes, I think more metadata is needed. when applying this decoration, lib/inspect.py has a funtion called _signature_from_callable, which failed to get the signature from function defined in Clojure.

[jjtolton]

@DogLooksGood you've run into a wall I've hit before. Python treats Clojure functions the same way it treats C functions. If you do

>>> import inspect
>>> import functools
>>> inspect.getfullargspec(functools.reduce)
Traceback (most recent call last):
  File "/usr/lib/python3.6/inspect.py", line 1126, in getfullargspec
    sigcls=Signature)
  File "/usr/lib/python3.6/inspect.py", line 2273, in _signature_from_callable
    skip_bound_arg=skip_bound_arg)
  File "/usr/lib/python3.6/inspect.py", line 2097, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function reduce>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/inspect.py", line 1132, in getfullargspec
    raise TypeError('unsupported callable') from ex
TypeError: unsupported callable

Same thing happens if you try to inspect a Clojure function. There's currently no workaround for this.

[jjtolton]

There is a workaround to your problem though!

# mypythoncode.py
@pymc.deterministic
def some_variable(p=x):
    return 0.5 * p + 0.25

(require '[libpython-clj.require :refer [require-python import-python]])
(require '[libpythonj-clj.python :as py :refer [py..]])
(require-python 'mypythoncode)

(def my-x 42)
(py.. mypythoncode/some_variable
    -__closure__
    (__getitem__ 0)
    -cell_contents
    (__setattr__ "__defaults__" [my-x]))

You could wrap this in a function, like,

(defn update-closure-defaults [pyfn defaults]
  (py.. pyfn
      -__closure__
      (__getitem__ 0)
      -cell_contents
      (__setattr__ "__defaults__" defaults)))

[DogLooksGood]

Wow, this is great! let me have a try!

[jjtolton]

For your reference:

In [32]: def foo(a=2):
    ...:     return a + a
    ...: 
    ...: 

In [33]: def d(f):
    ...:     def _d(*args, **kwargs):
    ...:         return f(*args, **kwargs) + 1
    ...:     return _d
    ...: 

In [34]: food = d(foo)

In [35]: food()
Out[35]: 5

In [36]: food.__closure__[0].cell_contents.__setattr__("__defaults__", (0,))

In [37]: food()
Out[37]: 1

[jjtolton]

Second, what ** in this python code, is overloaded as numpy's power I think. So is this approach suitable?

@DogLooksGood are you referring to ** as in 2 ** 5 or ** as in foo(**kwargs)?

[jjtolton]

(require-python '[operator :as op])
(op/pow 2 6)

and

(require-python '[libpython-clj.python :as py :refer [py**]])
(py** foo kwargs)

respectively

[DogLooksGood]

When trying this solution, run into an issue that random crash happen when I execute require-python.

After some investigation, I could provide more information.

[jjtolton]

Okay -- can you share details about your project setup? Or at least, take a look at this template and let me know how your project deviates from the template.

[cnuernber]

Closing this for now. @DogLooksGood - If you can reproduce the crash let us know. We have several issues on crashing especially where torch is involved so I would first check those out. This is really a documentation issue that is now popped up a couple times so perhaps we should have a document addressing this issue (decorators) specifically.