Use Cython directive binding=True to get signatures for cython methods

[kwankyu]

In requests for help for cythonized built-in methods, the signature of the method is not shown, unlike normal python methods. For an example,

sage: a=17 
sage: a.quo_rem?

Docstring:     
   Returns the quotient and the remainder of self divided by other.
   Note that the remainder returned is always either zero or of the
   same sign as other.

   INPUT:

   * "other" - the divisor

   OUTPUT:

   * "q" - the quotient of self/other

   * "r" - the remainder of self/other

   EXAMPLES:

      sage: z = Integer(231)
      sage: z.quo_rem(2)
      (115, 1)
...

To fix this, we set Cython directive binding=True. Thus we buy

• better support of introspection into cython methods,
• consistent help messages for cython methods like python methods,
• better tracebacks on exceptions,
• better behaving for documentation

for slight performance degradation due to increased overhead cost of calling cython methods.

Related tickets: #19100, #20860, #18192

Depends on #32509 Depends on #33864

CC: @jdemeyer @tscrim @mkoeppe

Component: user interface

Author: Kwankyu Lee, Tobias Diez

Branch/Commit: public/26254 @ 326f19c

Reviewer: Tobias Diez, ...

Issue created by migration from https://trac.sagemath.org/ticket/26254

[tobiasdiez]

comment:54

After merging the latest dev branch, this now throws

File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/all_cmdline.py", line 19, in <module>
    from sage.all import *
  File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/all.py", line 135, in <module>
    from sage.symbolic.all   import *
  File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/symbolic/all.py", line 8, in <module>
    import sage.symbolic.expression  # initialize pynac before .ring
  File "sage/symbolic/expression.pyx", line 1, in init sage.symbolic.expression
    # -*- coding: utf-8 -*-
  File "sage/symbolic/function.pyx", line 144, in init sage.symbolic.function
    from .expression import (
ImportError: cannot import name call_registered_function

when importing sage.all, see eg github build workflow. Any idea on how to fix it?

This method was introduced in #32386 (or one of its dependencies).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

0517de6

Merge remote-tracking branch 'origin/develop' into public/26254

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 52b5089 to 0517de6

[tobiasdiez]

comment:56

I tried to investigate this is a bit, but didn't get far.

With bindings=true, the expression.cpp file contains the new lines

...
static PyMethodDef __pyx_mdef_4sage_8symbolic_10expression_121call_registered_function = {"call_registered_function", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_121call_registered_function, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_120call_registered_function};
....
static PyMethodDef __pyx_mdef_4sage_8symbolic_10expression_123find_registered_function = {"find_registered_function", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_123find_registered_function, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_122find_registered_function};
....

However, it no longer contains

static PyMethodDef __pyx_methods[] = {
  {"is_Expression", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_79is_Expression, METH_O, __pyx_doc_4sage_8symbolic_10expression_78is_Expression},
  {"is_SymbolicEquation", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_81is_SymbolicEquation, METH_O, __pyx_doc_4sage_8symbolic_10expression_80is_SymbolicEquation},
  {"_is_SymbolicVariable", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_83_is_SymbolicVariable, METH_O, __pyx_doc_4sage_8symbolic_10expression_82_is_SymbolicVariable},
  {"_repr_Expression", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_91_repr_Expression, METH_O, __pyx_doc_4sage_8symbolic_10expression_90_repr_Expression},
  {"_latex_Expression", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_93_latex_Expression, METH_O, __pyx_doc_4sage_8symbolic_10expression_92_latex_Expression},
  {"new_Expression", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_99new_Expression, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_98new_Expression},
  {"new_Expression_from_pyobject", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_101new_Expression_from_pyobject, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_100new_Expression_from_pyobject},
  {"new_Expression_wild", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_103new_Expression_wild, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_102new_Expression_wild},
  {"new_Expression_symbol", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_105new_Expression_symbol, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_104new_Expression_symbol},
  {"print_order", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_107print_order, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_106print_order},
  {"print_sorted", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_109print_sorted, METH_O, __pyx_doc_4sage_8symbolic_10expression_108print_sorted},
  {"math_sorted", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_111math_sorted, METH_O, __pyx_doc_4sage_8symbolic_10expression_110math_sorted},
  {"mixed_order", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_113mixed_order, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_112mixed_order},
  {"mixed_sorted", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_115mixed_sorted, METH_O, __pyx_doc_4sage_8symbolic_10expression_114mixed_sorted},
  {"call_registered_function", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_121call_registered_function, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_120call_registered_function},
  {"find_registered_function", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_123find_registered_function, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_122find_registered_function},
  {"register_or_update_function", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_4sage_8symbolic_10expression_125register_or_update_function, METH_VARARGS|METH_KEYWORDS, __pyx_doc_4sage_8symbolic_10expression_124register_or_update_function},
  {"get_sfunction_from_serial", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_127get_sfunction_from_serial, METH_O, __pyx_doc_4sage_8symbolic_10expression_126get_sfunction_from_serial},
  {"get_sfunction_from_hash", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_129get_sfunction_from_hash, METH_O, __pyx_doc_4sage_8symbolic_10expression_128get_sfunction_from_hash},
  {"make_map", (PyCFunction)__pyx_pw_4sage_8symbolic_10expression_131make_map, METH_O, __pyx_doc_4sage_8symbolic_10expression_130make_map},
  {0, 0, 0, 0}
};

which are present with bindings=false. Maybe its related to https://github.com/cython/cython/issues/1658.

After replacing expression.cpp with the one from develop everything seems to work fine.

Does anyone has an idea how to properly fix this? I tried setting bindings=false for expression.pyx, but then the compilation doesn't succeed anymore.

[tobiasdiez]

comment:57

I took a few doctest runs as benchmarks, and could only find a very small negative impact (about 1%) if at all.

bindings=True

hyperfine -i --warmup 3 './sage -t --long src/sage/manifolds/differentiable/pseudo_riemannian.py' --show-output
  Time (mean ± σ):     13.183 s ±  0.266 s    [User: 14.732 s, System: 1.173 s]
  Range (min … max):   12.937 s … 13.730 s    10 runs

hyperfine -i --warmup 3 './sage -t --long src/sage/matrix/matrix0.pyx' --show-output
  Time (mean ± σ):      7.719 s ±  0.176 s    [User: 7.039 s, System: 0.918 s]
  Range (min … max):    7.472 s …  7.993 s    10 runs

develop

hyperfine -i --warmup 3 './sage -t --long src/sage/manifolds/differentiable/pseudo_riemannian.py' --show-output
  Time (mean ± σ):     13.096 s ±  0.199 s    [User: 14.618 s, System: 1.167 s]
  Range (min … max):   12.889 s … 13.524 s    10 runs

hyperfine -i --warmup 3 './sage -t --long src/sage/matrix/matrix0.pyx' --show-output
  Time (mean ± σ):      7.675 s ±  0.165 s    [User: 6.940 s, System: 0.970 s]
  Range (min … max):    7.364 s …  7.841 s    10 runs

(disclaimer: these were run on gitpod, so there is no guarantee that the same resources were available)

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

8d9bad0

Merge branch 'develop' into public/26254

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0517de6 to 8d9bad0

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

bea228b

Try with localized import

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 8d9bad0 to bea228b

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

89b8081	Inline all other imports from expression
9bbf574	Fix a few tests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bea228b to 9bbf574

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

803804f

and one more test

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9bbf574 to 803804f

[tobiasdiez]

comment:62

This is now mostly working for me. A few doctests are still failing, mostly around cached functions and latex expressions. For some reason, some of the TypeErrors are now IndexErrors complaining that the index is out of bounds. Any idea where this is coming from?

[tobiasdiez]

Attachment: Screenshot 2022-05-08 133226.jpg

[tobiasdiez]

comment:63

I tried to track down the source of the following errors:

      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/repl/display/pretty_print.py", line 144, in pretty
        ok = representation(obj, self, cycle)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/repl/display/fancy_repr.py", line 348, in __call__
        ascii_art_repr = ascii_art_repr or o.parent()._repr_option('element_ascii_art')
    IndexError: tuple index out of range

Debugging the following test script

import sage.all
from sage.combinat.root_system.cartan_type import CartanType
from sage.repl.rich_output import pretty_print

C = CartanType(["A", 3, 1])

class MyCartanType:
    def my_method(self):
        return "I am here!"

C._add_abstract_superclass(MyCartanType)
pretty_print(C.__class__)
# <class 'sage.combinat.root_system.type_A_affine.CartanType_with_superclass_with_superclass'>
pretty_print(C.__class__.__bases__)
# This triggers the index error, but should print
# (<class 'sage.combinat.root_system.type_A_affine.CartanType_with_superclass'>,
#    <class ...__main__.MyCartanType...>)

leads to the following output in fancy_repr.py, line 348:

For some reason calling parent throws the IndexError. The python debugger doesn't give more insights, and I have no experience with cython debugging to see where the indexerror is thrown. So it would be nice if someone with more cython experience can take this from here.

[kwankyu]

comment:64

Replying to @tobiasdiez:

After merging the latest dev branch, this now throws

File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/all_cmdline.py", line 19, in <module>
    from sage.all import *
  File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/all.py", line 135, in <module>
    from sage.symbolic.all   import *
  File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/symbolic/all.py", line 8, in <module>
    import sage.symbolic.expression  # initialize pynac before .ring
  File "sage/symbolic/expression.pyx", line 1, in init sage.symbolic.expression
    # -*- coding: utf-8 -*-
  File "sage/symbolic/function.pyx", line 144, in init sage.symbolic.function
    from .expression import (
ImportError: cannot import name call_registered_function

when importing sage.all, see eg github build workflow. Any idea on how to fix it?

This method was introduced in #32386 (or one of its dependencies).

It seems that cython fails to import the function call_registered_function from sage.symbolic.expression via the src/sage/symbolic/pynac_function_impl.pxi file, where it is defined. I am not sure if we could regard this as a bug of cython as .pxi files are not supposed to contain function definitions. I wonder why it is moved there in the first place. Anyway I think we should ask to cython community about this first before we refactor our code.

[kwankyu]

comment:65

Replying to @kwankyu:

Anyway I think we should ask to cython community about this first before we refactor our code.

I filed an issue:

https://github.com/cython/cython/issues/4775

expecting any input from cython experts.

[tobiasdiez]

comment:66

Thanks for opening the issue.

Do you have an idea where the IndexError is coming from?

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

a23ffe9

Tiny edits

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 803804f to a23ffe9

[kwankyu]

comment:68

Replying to @kwankyu:

Replying to @kwankyu:

Anyway I think we should ask to cython community about this first before we refactor our code.

I filed an issue:

https://github.com/cython/cython/issues/4775

expecting any input from cython experts.

According to the answer, there was a circular import that somehow was hidden but revealed only with binding=True. Hence I think changing global imports to local imports are the right solution.

[kwankyu]

comment:69

Replying to @tobiasdiez:

Do you have an idea where the IndexError is coming from?

Note that with binding=False:

class NewClass:
    def parent(self):
        return 1

NewClass.parent()
...
TypeError: parent() missing 1 required positional argument: 'self'

%%cython
cdef class NewClass:
    def parent(self):
        return 1

NewClass.parent()
...
TypeError: unbound method NewClass.parent() needs an argument

With binding=True:

%%cython
cimport cython
@cython.binding(True)
cdef class NewClass:
    def parent(self):
        return 1

NewClass.parent()
...
IndexError: tuple index out of range

NewClass.parent(2)
1

In the latter, NewClass is of course not an object but a class, hence NewClass.parent() invokes unbound method .parent which is a cython function that thinks it is bound. Hence it tries to get "self" from its dictionary of arguments. As the dictionary is empty, it raises IndexError. I think this is a bug of cython. It should raise TypeError as it should follow Python behavior.

Cython issue:

https://github.com/cython/cython/issues/4779

[kwankyu]

comment:70

Replying to @kwankyu:

Cython issue:

https://github.com/cython/cython/issues/4779

They are fixing this bug. I think we should wait for the new Cython instead of "fixing it" on our side.

[kwankyu]

comment:71

Replying to @tobiasdiez:

I took a few doctest runs as benchmarks, and could only find a very small negative impact (about 1%) if at all.

Now I think we should regard this 1% as our debt (it was not ours), or as something to pay to get proper introspection to cython functions, which weighs more than the 1%.

Perhaps we may selectively turn on "binding=False" for some cython functions critical for performance.

[tobiasdiez]

comment:72

Replying to @kwankyu:

Replying to @tobiasdiez:

Do you have an idea where the IndexError is coming from?

Note that with binding=False:

class NewClass:
    def parent(self):
        return 1

NewClass.parent()
...
TypeError: parent() missing 1 required positional argument: 'self'

%%cython
cdef class NewClass:
    def parent(self):
        return 1

NewClass.parent()
...
TypeError: unbound method NewClass.parent() needs an argument

With binding=True:

%%cython
cimport cython
@cython.binding(True)
cdef class NewClass:
    def parent(self):
        return 1

NewClass.parent()
...
IndexError: tuple index out of range

NewClass.parent(2)
1

In the latter, NewClass is of course not an object but a class, hence NewClass.parent() invokes unbound method .parent which is a cython function that thinks it is bound. Hence it tries to get "self" from its dictionary of arguments. As the dictionary is empty, it raises IndexError. I think this is a bug of cython. It should raise TypeError as it should follow Python behavior.

Cython issue:

https://github.com/cython/cython/issues/4779

Nice! Thanks for doing the analysis and finding a nice minimal example reproducing the issue.

[tobiasdiez]

comment:73

Should be fixed in the new cython release. Updating to it is tracked at #33864.

[tobiasdiez]

Changed dependencies from #32509 to #32509, #33864

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from a23ffe9 to 570058c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

92e9cff	build/pkgs/cython: Update to 0.29.30
570058c	Merge remote-tracking branch 'origin/u/mkoeppe/update_cython_to_0_29_29' into public/26254

[tobiasdiez]

comment:75

Seems like the update of cython to 0.29.30 didn't help, the index errors are still present (at least in the github workflow).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

02d6282

Merge remote-tracking branch 'origin/develop' into public/26254

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 570058c to 02d6282

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 02d6282 to f5bc80d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9d94b02	Merge remote-tracking branch 'origin/develop' into public/26254
f5bc80d	Fix more tests

[tobiasdiez]

comment:78

This is mostly working now. There are a few failing doctests related to pickling, and what I believe is connected, module names (see the build & test workflow). Moreover, a couple of doctests fail in pynac_impl.pxi. I sadly don't have the expertise to fix these (on my own) and would appreciate input from cython experts.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f5bc80d to f38dc83

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f38dc83

Set cython directive binding=True

[kwankyu]

comment:81

This is working now. While I am not a cython expert, I managed to fix remaining failing doctests. Setting "needs review" to test.

[kwankyu]

comment:82

To break circular imports, it is now not possible to import

    call_registered_function, 
    find_registered_function, 
    register_or_update_function,
    get_sfunction_from_hash,
    get_sfunction_from_serial

from sage.symbolic.function. They should be imported from sage.symbolic.expression, where they are defined.

But then I could not figure out how to deprecate old imports. Or it would be better to make them importable from sage.symbolic.function as well. But I couldn't figure out how to do this either!

[tobiasdiez]

comment:84

Thanks, nice that its working now.

[tobiasdiez]

Changed author from Kwankyu Lee to Kwankyu Lee, Tobias Diez

[tobiasdiez]

Reviewer: Tobias Diez, ...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f38dc83 to 80375a2

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

80375a2

Put symbolic function related functions back into sage.symbolic.function

[kwankyu]

comment:86

The last commit allows importing

    call_registered_function, 
    find_registered_function, 
    register_or_update_function,
    get_sfunction_from_hash,
    get_sfunction_from_serial

from sage.symbolic.function.

[jhpalmieri]

comment:87

I think that Cython has to be rebuilt for this to take effect. (At least, I got some doctest failures when I built by just using the branch, and they went away after doing make distclean and trying again.) Can you modify the branch to trigger that, for example by bumping up the patch level on the Cython version number?

[kwankyu]

comment:88

Replying to John Palmieri:

I think that Cython has to be rebuilt for this to take effect.

Perhaps no for Cython itself, but yes all cython files must be rebuilt.

Can you modify the branch to trigger that, for example by bumping up the patch level on the Cython version number?

Okay.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 80375a2 to 735fbcf

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

735fbcf

Bump up patch level for Cython