More documentation on how to incorporate extension modules into a package

[peekxc]

I would like to use pythran for a project, but the docs aren't complete enough for my to reasonably figure out how many parts on how to incorporate it outside of calling the CLI manually.

It would be great if Pythran had a minimal reproducible example showing, for example:

• How to build a package with extension modules using Pythran + setuptools
• How to build a package with extension modules using Pythran + meson-python
• Various often-used options in .pythranrc
• How to specify / link different BLAS libs
• How to do the above with .pythran files
• How to link openmp with the generated code
• etc etc

In the docs it very briefly, but I don't think it's nearly enough. I'm familiar with meson + meson-python for building extension modules, but after about an hour of work I still haven't figured out how to get a single extension module transpiled to C++ and built into a wheel.

I know it says "see SciPy for an example of doing this with Meson", but SciPy is simply too large of a project to use a reference (indeed, I would say SciPy is among the most complicated libraries in terms of build complexity in the entire scientific Python ecosystem!).

[paugier]

For simpler examples with Meson + Pythran, see

• https://foss.heptapod.net/fluiddyn/fluidimage
• https://foss.heptapod.net/fluiddyn/fluidfft
• https://foss.heptapod.net/fluiddyn/fluidsim

[peekxc]

For simpler examples with Meson + Pythran, see

• https://foss.heptapod.net/fluiddyn/fluidimage
• https://foss.heptapod.net/fluiddyn/fluidfft
• https://foss.heptapod.net/fluiddyn/fluidsim

Thanks for the reply, however these projects are not great examples of what I would want ('a typical way to use pythran with e.g. meson-python'), as they use some custom backend calls to something called 'transonic', e.g.

run_command(['transonic', '--meson', '--backend', backend, 'operators.py'], check: true)

I just want an example which:

• Takes a pythan file name and transpiles it to C++ w/ Pythran
• Creates an extension module from said file

I cannot figure out how to do even these two things unfortunately in such a way that incorporates the pythran RC options using meson python

[peekxc]

Ahh, I believe I've figured it out (though I think the issue is still valid).

The key was some of the arguments that SciPy uses. IN particular, in the top-level meson.build, they have something like:

pythran = find_program('pythran', native: true, version: '>=0.14.0')
_cpp_args = [
    '-DENABLE_PYTHON_MODULE',
    '-D__PYTHRAN__=3'
]

For unknown reasons, the above macro variables seem important for creating extension modules with Pythran, as otherwise I ran into dynamic import errors / functions not exporting in their corresponding .so files.

The actual build step that worked for me was something like:

# Configure BLAS Libs
openblas_inc = include_directories('/usr/local/opt/openblas/include')
openblas_lib = '/usr/local/opt/openblas/lib'
openblas_link_args = ['-L' + openblas_lib, '-lopenblas']

# Transpile the .py files 
run_command(['pythran', '-E', 'hello_world.py', '-o', '_hello_world.cpp'], check: true)

# Create the extension modules
py_extension = py.extension_module(
    '_hello_world', 
    '_hello_world.cpp',
    include_directories: [openblas_inc],
    link_args: openblas_link_args,
    cpp_args: [_cpp_args],
    dependencies: [pythran_dep, np_dep],
    install: true, 
    subdir: 'my_package'
)

This exports a separate python module with the Pythran-transpilation, but I suppose one would instead probably replace the module hello_world in practice.

[serge-sans-paille]

Would you be okay to submit a PR which siummarize your understanding as an addition to the doc, and then we can iterate on this in a joint effort to improve the doc? That would be much appreciated :-)

[paugier]

Scikit-image (https://github.com/scikit-image/scikit-image) uses Pythran without Transonic and is much simpler than Scipy, but it also uses Cython. I don't know if there are other simpler packages that use Pythran without Transonic.

The simplest way to use Pythran with Meson in a Python package is by using Transonic, which takes care of writing the Pythran files and most Meson files. Only the root Meson file (example) contains Pythran specific code (basically the same code as for projects using Pythran directly without Transonic).

For each directory with Pythran files, there is a meson.build file with something like (automatically produced when using Transonic, hand written otherwise)

operators2d = custom_target(
  'operators2d',
  output: ['operators2d.cpp'],
  input: 'operators2d.py',
  command: [pythran, '-E', '--config', 'pythran.complex_hook=pythran_complex_hook', '@INPUT@', '-o', '@OUTDIR@/operators2d.cpp'],
  env: ['PYTHRANRC='],
)

operators2d = py.extension_module(
  'operators2d',
  operators2d,
  cpp_args: cpp_args_pythran,
  dependencies: [pythran_dep, np_dep],
  # link_args: version_link_args,
  install: true,
  subdir: 'fluidsim/operators/__pythran__',
)

operators3d = custom_target(
  'operators3d',
  output: ['operators3d.cpp'],
  input: 'operators3d.py',
  command: [pythran, '-E', '--config', 'pythran.complex_hook=pythran_complex_hook', '@INPUT@', '-o', '@OUTDIR@/operators3d.cpp'],
  env: ['PYTHRANRC='],
)

operators3d = py.extension_module(
  'operators3d',
  operators3d,
  cpp_args: cpp_args_pythran,
  dependencies: [pythran_dep, np_dep],
  # link_args: version_link_args,
  install: true,
  subdir: 'fluidsim/operators/__pythran__',
)

[serge-sans-paille]

@peekxc I've submited a minimal meson file that's tested by CI, and pointed to it in the doc , cf. https://pythran.readthedocs.io/en/latest/MANUAL.html#meson-integration

[serge-sans-paille]

Let's consider the main issue (meson integration) documented. Please submit individual issues or (better!) PR with better documentation ;-)