NNPDF / eko

Evolution Kernel Operators
https://eko.readthedocs.io
GNU General Public License v3.0
6 stars 2 forks source link

Use Cython as numba alternative #266

Closed felixhekhorn closed 7 months ago

felixhekhorn commented 1 year ago

related to #185 - for the moment I use this issue to save my work on a different place than just my computer ... eventually, I think, I can turn this into an PR.

at the moment it seems doing just poetry build is not enough, but doing just before a cythonize -X language_level=3 -a -i myfile.pyx seems to be sufficient

cc @scarrazza

Resources

Files

`pyproject.toml` ```toml [tool.poetry] name = "pkgcy" version = "0.0.0" description = "Blubbes" authors = ["Lorem Ipsum "] license = "MIT" keywords = [] packages = [ { include = "pkgcy" }, # { include = "tests", format = "sdist" }, ] include = [ # { path = "meson.build", format = "sdist" }, # { path = "py.typed" }, # C extensions must be included in the wheel distributions { path = "*.so", format = "wheel" }, { path = "*.pyd", format = "wheel" }, # for Windows ] [tool.poetry.dependencies] python = "^3.8,<3.12" numpy = "^1.24" scipy = "^1.10.1" [tool.poetry.group.dev.dependencies] cython = ">=3.0.0b2" [tool.poetry.build] generate-setup-file = false script = "mybuild.py" # the name build.py is taken by poetry itself [build-system] requires = ["poetry-core>=1.1.0a6", "Cython>=3.0.0b2"] # or >= 0.29? build-backend = "poetry.core.masonry.api" ```
`pkgcy/__init__.py` ```py from scipy.integrate import quad import os if os.environ.get("BUILD_WHEEL",False): pass else: import pyximport; pyximport.install() from myfile import f as g def f(x): return x def run(): return quad(g, 0.,1.) ```

`mybuild.py` ```py import os # See if Cython is installed try: from Cython.Build import cythonize # Do nothing if Cython is not available except ImportError: # Got to provide this function. Otherwise, poetry will fail def build(setup_kwargs): pass # Cython is installed. Compile else: # from setuptools import Extension # from setuptools.dist import Distribution from distutils.command.build_ext import build_ext # This function will be executed in setup.py: def build(setup_kwargs): # The file you want to compile extensions = [ "myfile.pyx" ] # gcc arguments hack: enable optimizations os.environ['CFLAGS'] = '-O3' # Build setup_kwargs.update({ 'ext_modules': cythonize( extensions, language_level=3, compiler_directives={'linetrace': True}, ), 'cmdclass': {'build_ext': build_ext} }) ```

`myfile.pyx` ```cython # cython: language_level=3 def f(x): return x*x*x ```
felixhekhorn commented 1 year ago

Cython and numba:

Cython submodules:

myfilea.pyx ```cython # cython: language_level=3 from src2.myfile2 import f2 cpdef api double foo(double x): return x * f2(x) ```
src2/myfile2.pyx ```cython # cython: language_level=3 def f2(x): return 2.*x*x ```
pkgcy/__init__.py ```python from scipy.integrate import quad import os # if os.environ.get("BUILD_WHEEL",False): # pass # else: # import pyximport; pyximport.install() # from myfilea import f3a as g # # import myfile import numba as nb import ctypes from numba.extending import get_cython_function_address addr = get_cython_function_address("myfilea", "foo") functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double) myexp = functype(addr) @nb.njit def f(x): return x * myexp(x) def run(): # print(myfile.__pyx_capi__) # return f(10.) return quad(f, 0.,1.) ```

and

alecandido commented 1 year ago

We can call more or less in the same way any C library from Numba:

https://numba.readthedocs.io/en/stable/user/cfunc.html#calling-c-code-from-numba

At that point, I'd write the code directly in C/C++, or in Rust and providing a C API. Moreover, providing the C library for external users was the goal since the beginning. If EKO consumes the same one, at least during the transition, it's not bad (eventually, when the top-level infrastructure will move out of Numba, we could also consume a different kind of API). In any case, we aim to drop Numba.

felixhekhorn commented 1 year ago

yes, but the main advantage of using Cython is the minimal effort in rewriting, because I'm still afraid of doing all in one gigantic step ... this way we could, maybe, do step by step, though I'm not sure still ...

alecandido commented 1 year ago

this way we could, maybe, do step by step, though I'm not sure still...

This is exactly what I was pointing out: the same feature that allows Cython to be introduced one step at a time could be used with whatever C library (because it's based on ctypes).

felixhekhorn commented 1 year ago

Cython is the minimal effort in rewriting

well, we would still need to rewrite everything from scratch (using C/Rust syntax)

alecandido commented 1 year ago

well, we would still need to rewrite everything from scratch (using C/Rust syntax)

For as long as it's just math, the C/Rust syntax is mostly Python syntax. It was Fortran in the first place... (and doing it incrementally should make this a fully negligible concern)

felixhekhorn commented 7 months ago

closed in favour of #189