Allow calling external cythonized functions from pythranized functions

[ashwinvis]

Maybe I am really naive, and this is not the way it should be done. In some sense this is related to #461.

Background

Here we show some attempts to implement a function which performs ($$\vec{u}. \nabla \vec{u_fft}$$) through a pseudospectral approach. u is a 3D vector in x, y, z and u_fft is its spectral counterpart in k_x, k_y, k_z. The gradient operator $$\nabla \vec{u}$$) is calculated as $$ ifft(1j . \vec{k} (u_fft)) $$. Where ifft means an inverse FFT operation and wavenumber vector k = (k_x, k_y,k_z)

Problem

Enough math. Let's talk code now. Take a look at how we tried to implement this (note: ifft3d is a cythonized function)

• Pythran way: https://github.com/fluiddyn/fluidfft/blob/master/fluidfft/fft3d/dream_pythran.py
• Pythran + Cython way: https://github.com/fluiddyn/fluidfft/blob/master/fluidfft/fft3d/dream_cythran.pyx

We would like to implement the rest of the operations using Pythran. We know the exact input and return types of ifft3d. However, we have upto 5 different implementations of ifft3d depending on which FFT backend is specified / available.

I have tried my best to describe our attempts. If needed I can try to create a simpler test case.

Is it possible for pythran to access a cythonized function?

[serge-sans-paille]

MMMh sounds very interesting. It is currently not possible but that's in my plan. We need to do two things there:

1. allow pythran to accept c function pointer as parameter, e.g. though a PyCapsule
2. Put a cython generated function into a capsule

Note that since 357d4d8e94071dc860afd1a8 it's already possible to put a pythran function in a capsule.

I think @paugier is liklely to be interested in that feature!

Note about the pythran annotation: what should be the pereferd way to describe a capsule containing a function pointer from (int, float) to int?

int(int, float)  # cstyle
int -> float -> int # ml style
(int, float) -> int # ml + python style ?

[serge-sans-paille]

For the cython part, a lookup into __pyx_capi__ seems to be enough, great!

[ashwinvis]

Of course, @paugier and me are working together on this. I did read your "Capsule corporation" blog post. Just to be clear, you plan to pass a cython generated function as an argument though a PyCapsule and that needs to be implemented, right?

EDIT: Should the PyCapsule part be done externally, or will pythran takes care of this?

On the pythran annotation question: (personally) I would prefer either the "cstyle" (cython users might like that) or the "ml + python" style. "ml" style looks too verbose

[serge-sans-paille]

ok, I'll give a try to the cstyle one, that clearly indicates that the fnction must come from C and is not some kind of Python function.

[serge-sans-paille]

see #746 for a not so tested implementation. Turns out it was relatively simple to implement, thanks to pythran magical typing system :-)

[serge-sans-paille]

cc @paugier btw

[ashwinvis]

Yay for the implementation. An example / test would be nice. I tried to use it as follows.

1. Pythranized a file called pythran/tests/cython/add2.py

   
   # pythran export add2(
   #     float[][], float[][], float[][](float[][], float[][]))

def add2(f, g, add): return add(f, g)

2. Built the cython extension `pythran/tests/cython/add.py` using `setup_add.py`
3. Test!
```python
from add import add
from add2 import add2
import numpy as np

a = np.ones([3, 3])
b = a * 3
ans = add(a, b)
ans2 = add2(a, b, add)

Obviously I am doing it wrong:

TypeError: Invalid argument type for pythranized function `add2'.
Candidates are:
   add2(ndarray<double,2>,ndarray<double,2>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)
   add2(ndarray<double,2>,numpy_texpr<ndarray<double,2>>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)
   add2(numpy_texpr<ndarray<double,2>>,ndarray<double,2>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)
   add2(numpy_texpr<ndarray<double,2>>,numpy_texpr<ndarray<double,2>>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)

[serge-sans-paille]

It only accepts capsule as first parameter, so you should#pythran export capsule add2

[ashwinvis]

So now I changed add2.py into

# pythran export capsule add1(float[][], float[][])
def add1(f, g):
    return f + g

# pythran export add2(
#     float[][], float[][], float[][](float[][], float[][]))
def add2(f, g, add_func):
    return add_func(f, g)

In [1]: from add2 import add1, add2
In [2]: import numpy as np
In [3]: a = np.ones([3, 3])
In [4]: b = a * 3
In [5]: add2(a, b, add1)
Out[5]: 
array([[ 4.,  4.,  4.],
       [ 4.,  4.,  4.],
       [ 4.,  4.,  4.]])

If only add_func could be a Cython function automagically translated into a PyCapsule! We won't need scipy.LowLevelCallable. (EDIT: excuse my ignorance!)

[serge-sans-paille]

@ashwinvis works out of the box, check the following:

cython file:

cdef api add(int x, int y):
    cdef int r;
    r = x + y;
    return r

ipython sessions:

>>>  import add
>>>  add.__pyx_capi__['add']
<capsule object "PyObject *(int, int)" at 0x7f27fe3587e0>

[ashwinvis]

It runs but is buggy. Can you have a look at https://github.com/ashwinvis/cython_capi/tree/master/using_api_keyword?

Clone and simply run inside the directory:

make build
make test

This is what I get by adding 1 and 2

======================================================================
FAIL: test_int (__main__.TestAll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 14, in test_int
    np.testing.assert_equal(ans, ans2)
  File "/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/testing/utils.py", line 416, in assert_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
 ACTUAL: 3
 DESIRED: 94688066438504

[ashwinvis]

There is some hope though. It could be possible that Cython's api keyword could be buggy. See this simple case (which works!): https://github.com/ashwinvis/cython_capi/tree/master/using_pxd

[serge-sans-paille]

Yeah, you need to make sure the function registered in the capsule actually takes the expected parameters. Feel free to open a bug on cython :-)

[serge-sans-paille]

@ashwinvis what you wanted to do was actually:

cdef api int add(int x, int y):
    cdef int r;
    r = x + y;
    return r

forcing the return type gives you the right signature (here int (int, int))

[ashwinvis]

You are right. It even works with arrays! Now I need to try this in FluidFFT.

-------- Original Message -------- On 27 Nov 2017, 21:42, serge-sans-paille wrote:

@ashwinvis what you wanted to do was actually:

cdef api

int

add(

int

x,

int

y): cdef

int

r

;

r

=

x

+

y

;

return

r

forcing the return type gives you the right signature (here int (int, int))

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[serge-sans-paille]

Pythran part implemented in #746