@vectorize-decorated functions are not found by sphinx

[HPLegion]

• [x] I am using the latest released version of Numba (most recent is visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
• [ ] I have included below a minimal working reproducer (if you are unsure how to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

I use sphinx with the apidoc extension to have a reference documentation generated automatically. For functions decorated with njit this works without any complications, for vectorize however, the function disappears from the docs, indicating that apidoc either cannot find it or decides not to include it. (Sorry for the lack of better info).

Since this is sphinx related I am not sure how to include a minimal example, but if it would help and I find some time I can try to set up a mini-repository.

[esc]

@HPLegion thanks for submitting this. I'm afraid, w/o a reproducer this will be difficult to triage. Perhaps you could make a patch against the Numba source tree (which also uses sphinx for documentation) to expose the issue? Or if you have this in an open-source project, perhaps link to it? Perhaps you could try messing with different versions of sphinx/apidoc and see if the issue vanishes? Sorry I don't have any better ideas here.

[HPLegion]

I have gotten around to construct a small example #5765. In that process I also realised that the problem does not lie with sphinx apidoc (which just creates the skeleton) but with autodoc, which generates the actual reference documentation.

[HPLegion]

Okay, after stepping through sphinx I think I narrowed it down to the following: njit functions have the __wrapped__ attribute which I suppose contains the original python function. Sphinx can resolve this. This field does not exist for vectorize, hence autodoc thinks that it cannot document the function and skips it (it is not rejected based on any filters as far as I can tell, autodoc just gives up)

EDIT This is where the dispatcher updates the wrapper information https://github.com/numba/numba/blob/70f7b687bf7bc25a025a71aed5d74551f380fa5a/numba/core/dispatcher.py#L679 A similar step seems to happen for cfuncs https://github.com/numba/numba/blob/5f474010f8f50b3cf358125ba279d345ae5914ef/numba/core/ccallback.py#L45 But not for DUFuncs https://github.com/numba/numba/blob/5f474010f8f50b3cf358125ba279d345ae5914ef/numba/np/ufunc/dufunc.py#L79-L85

[HPLegion]

Some more digging has revealed that njit functions get documented by autodoc because

• they are a methoddescriptor
• & have the python function in __wrapped__
• & have the correct __module__ string.

For njit all these fields are set through the call to functools.update_wrapper, and the __get__ but not __set__ methods are implemented which qualifies them as methoddescriptors.

>>> from numba import njit
>>> import inspect
>>> @njit
... def f(x):
...     return x
... 
>>> f(1)
1
>>> inspect.isfunction(f)
False
>>> inspect.ismethoddescriptor(f)
True
>>> inspect.isroutine(f)
True
>>> hasattr(f, "__wrapped__")
True
>>> f.__module__
'__main__'
>>> f.__qualname__
'f'
>>> f.__wrapped__
<function f at 0x7f0b87558830>

>>> from numba import vectorize
>>> import inspect
>>> @vectorize
... def g(x):
...     return x
... 
>>> g(1)
1
>>> inspect.isfunction(g)
False
>>> inspect.ismethoddescriptor(g)
False
>>> inspect.isroutine(g)
False
>>> hasattr(g, "__wrapped__")
False
>>> g.__module__
'numba.np.ufunc.dufunc'
>>> g.__qualname__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'DUFunc' object has no attribute '__qualname__'
>>> g.__wrapped__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'DUFunc' object has no attribute '__wrapped__'

[esc]

@HPLegion good to see you are getting somewhere investigating this. Do you think there might be a way to adapt this technique for @vectorize too?

[HPLegion]

Yes, I think so.

TL;DR: I tried and numba hasn't exploded

A few thoughts

• I was scratching my head on whether the dispatchers should appear to be functions, so I went and checked out lru_cache from functools to get a similar wrapper-example from the standard library. It turns out that functions decorated with lru_cache do not pass the isfunction check, they are indeed also methoddescriptors - so being a methoddescriptor seems like a reasonable choice for a decorated function
• As expected lru_cache uses update_wrapper to set its own fields to the appropriate values. It is hence very much comparable to the jit Dispatcher and both of them seem to get documented just fine
• I imagine (without any actual arguments) that it may generally be advantageous (not just for sphinx) if numba's decorators are relatively transparent and follow what seems to be the conventions of filling the respective __magic__ attributes
• Ideally one would use update_wrapper for a clean and standardized propagation of the decorated function attributes and implement __get__ but not __set__ for vectorize (and maybe even further dispatchers), but this raises some questions
  • Are there downsides to being a methoddescriptor? Would there be a promise about certain functionality that cannot be upheld
  • update_wrapper sets a number of attributes, but by default also updates the entire __dict__, this smells like a possibility for unforseen consequences (but it could be disabled)
  • Are any of ('__module__', '__name__', '__qualname__', '__doc__', '__annotations__') crticial within numba and could break it if they were changed.

Experiment

Yesterday I set up a small branch where I simply called update_wrapper(self, py_func) in the DUFunc constructor and copied the __get__ implementation from the jit Dispatcher. As far as I can tell that resulted in sphinx succesfully documenting vectorize-decorated functions, but I did not stress test it. There seems to be a plethora of Dispatchers for different targets and different ways to construct them so I am sure that this simple tweak did not fix all of them. However the good news is that it does not seem like I broke much in the process; all tests in tests/npyufunc passed (I hope that was the right test set to focus on :stuck_out_tongue: )

References

lru_cache calling update_wrapper https://github.com/python/cpython/blob/4649202ea75d48e1496e99911709824ca2d3170e/Lib/functools.py#L765

update_wrapper https://github.com/python/cpython/blob/4649202ea75d48e1496e99911709824ca2d3170e/Lib/functools.py#L33-L50

[HPLegion]

I had some time yesterday and found myself thinking that bending the inner workings of numba for the sake of sphinx may not be the way to go (even though there may or may not be other advantages to this). So I had a bit of a look for ways to fixing my problem and got inspired by celery - they are offering a small sphinx extension as part of their package to allow auto-documentation of Celery (decorated) tasks.

The advantage to such an approach could be that it actually offers the possibility to customise the documentation behaviour. I hacked together a proof of concept, and it is not too terrifying, luckily Sphinx has a fairly comfortable extension API (but debugging event-driven programs turns out to be a nightmare :fearful:).

Here is a plaintext example of what could be done in principle. The documenter (see below) adds a prefix (@numba.vectorize) to the signature and injects information about the precompiled signature into the docstring.

@numba.vectorize ebisim.plasma.clog_ei(Ni, Ne, kbTi, kbTe, Ai, qi)

      *Precompiled signatures:[‘dddddl->d’]*

   The coulomb logarithm for ion electron collisions.

   Parameters:
      * **Ni** (*float** or **numpy.ndarray*) – <1/m^3> Ion density.

      * **Ne** (*float** or **numpy.ndarray*) – <1/m^3> Electron
        density.

      * **kbTi** (*float** or **numpy.ndarray*) – <eV> Ion
        temperature.

      * **kbTe** (*float** or **numpy.ndarray*) – <eV> Electron
        temperature.

      * **Ai** (*float** or **numpy.ndarray*) – Ion mass number.

      * **qi** (*int** or **numpy.ndarray*) – Ion charge state.

   Returns:
      Ion electron coulomb logarithm.

   Return type:
      float or numpy.ndarray

Is an extension like this something that could be interesting to feature in numba, or is it just too likely to become a maintenance liability? :stuck_out_tongue: (The code as it is right now would likely need some bulletproofing.)

From my side this issue can be closed, since I found a solution that is working for me but if you want to keep this open to track the feature request, that's fine.

Extension code

"""
This sphinx extension aims to improve the documentation of numba-decorated
functions. It it inspired by the design of Celery's sphinx extension
'celery.contrib.sphinx'.

Usage
-----

Add the extension to your :file:`conf.py` configuration module:

.. code-block:: python

    extensions = (...,
                  'numbadoc')

This extension adds two configuration fields, which determine the prefix
printed before jitted functions in the reference documentation.
Overwrite these values in your :file:`conf.py` to change this behaviour

.. code-block:: python

    #defaults
    numba_jit_prefix = '@numba.jit'
    numba_vectorize_prefix = '@numba.vectorize'

With the extension installed `autodoc` will automatically find
numba decorated objects and generate the correct docs.

If a vecotrized function with fixed signatures is found, these are injected
into the docstring.
"""
from typing import List, Iterator
from sphinx.domains.python import PyFunction
from sphinx.ext.autodoc import FunctionDocumenter

from numba.core.dispatcher import Dispatcher
from numba.np.ufunc.dufunc import DUFunc

class NumbaFunctionDocumenter(FunctionDocumenter):
    """Document numba decorated functions."""

    def import_object(self) -> bool:
        """Import the object given by *self.modname* and *self.objpath* and set
        it as *self.object*.

        Returns True if successful, False if an error occurred.
        """
        success = super().import_object()
        if success:
            # Store away numba wrapper
            self.jitobj = self.object
            # And bend references to underlying python function
            if hasattr(self.object, "py_func"):
                self.object = self.object.py_func
            elif hasattr(self.object, "_dispatcher") and \
                 hasattr(self.object._dispatcher, "py_func"):
                self.object = self.object._dispatcher.py_func
            else:
                success = False
        return success

    def process_doc(self, docstrings: List[List[str]]) -> Iterator[str]:
        """Let the user process the docstrings before adding them."""
        # Essentially copied from FunctionDocumenter
        for docstringlines in docstrings:
            if self.env.app:
                # let extensions preprocess docstrings
                # need to manually set 'what' to FunctionDocumenter.objtype
                # to not confuse preprocessors like napoleon with an objtype
                # that they don't know
                self.env.app.emit('autodoc-process-docstring',
                                  FunctionDocumenter.objtype,
                                  self.fullname, self.object,
                                  self.options, docstringlines)
            # This block inserts information about precompiled signatures
            # if this is a precompiled vectorized function
            if getattr(self.jitobj, "types", []) and \
               getattr(self.jitobj, "_frozen", False):
                s = "| *Precompiled signatures:" + str(self.jitobj.types) + "*"
                docstringlines.insert(0, s)
            yield from docstringlines

class JitDocumenter(NumbaFunctionDocumenter):
    """Document jit/njit decorated functions."""

    objtype = 'jitfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return isinstance(member, Dispatcher) and hasattr(member, 'py_func')

class VectorizeDocumenter(NumbaFunctionDocumenter):
    """Document vectorize decorated functions."""

    objtype = 'vecfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return isinstance(member, DUFunc) and \
               hasattr(member, '_dispatcher') and \
               hasattr(member._dispatcher, 'py_func')

class JitDirective(PyFunction):
    """Sphinx jitfun directive."""

    def get_signature_prefix(self, sig):
        return self.env.config.numba_jit_prefix

class VectorizeDirective(PyFunction):
    """Sphinx vecfun directive."""

    def get_signature_prefix(self, sig):
        return self.env.config.numba_vectorize_prefix

def setup(app):
    """Setup Sphinx extension."""
    # Register the new documenters and directives (autojitfun, autovecfun)
    # Set the default prefix which is printed in front of the function signature
    app.setup_extension('sphinx.ext.autodoc')
    app.add_autodocumenter(JitDocumenter)
    app.add_directive_to_domain('py', 'jitfun', JitDirective)
    app.add_config_value('numba_jit_prefix', '@numba.jit', True)
    app.add_autodocumenter(VectorizeDocumenter)
    app.add_directive_to_domain('py', 'vecfun', VectorizeDirective)
    app.add_config_value('numba_vectorize_prefix', '@numba.vectorize', True)

    return {
        'parallel_read_safe': True
    }

(ping @esc )

[soleti]

Hi, thank you very much for this extension, but my decorated functions still don't appear in the docs. The only change I made is using @cuda.jit instead of @numba.jit with this line in conf.py:

numba_jit_prefix = '@cuda.jit'

Do you have any idea why it doesn't work in this case?

[HPLegion]

Hej @soleti,

it's been a few days since I bashed this together, but my best guess at the moment is the following:

The Documenter classes have a method can_document_member which sphinx uses to determine the extensions ability to deal with certain objects. As they are written now, they probably simply return False for CUDA jit functions.

I see two possiblefixes here:

• Either copy the jit-documenter stuff and adjust it for cuda (that way you can also tweak how certain things get documented, if there is any additional things you want to have written down for cuda stuff)
• Or try to hack the jit-documenter into claiming it can document CUDA-jit functions (but since I don't know the CUDA part of numba I don't know if the tweaks I used for normal jit functions work there)

Let me know if that helps, and if yes, feel free to share your results here for future reference (if I have time one day, I may turn this into a tiny package, but more likely I will just document it in a better place (i.e. Luk's How-to-numba-repo))

[soleti]

I tried adapting the jit-documenter stuff but I couldn't get it to work. I adopted a solution that I am ashamed of but that works: running this before sphinx :)

find mypackage/ -name '*.py' -exec sed -i -e 's/@cuda/#@cuda/g' {} \;

[HPLegion]

Hi @soleti,

For me the following seems to work, feel welcome to give it a test spin and if yes expand on it.

(full disclaimer: I don't really know numba.cuda at all, so I am not aware if my check for CUDA funcs is even remotely sufficient or what nice gimmicks would be for documenting CUDA functions, also my Laptop has no GPU, so I only checked this without ever actually triggering the compilation of the CUDA decorated function)

---

Same sphinx extension as before with added cuda features: (You need to make sure that Sphinx loads this, by both adding it to the conf.py and making sure that it is actually on the current path for sphinx to find the module and import it, I called the file numbadoc.py (hence the suggested extension name in the docstring)

"""
This sphinx extension aims to improve the documentation of numba-decorated
functions. It it inspired by the design of Celery's sphinx extension
'celery.contrib.sphinx'.

Usage
-----

Add the extension to your :file:`conf.py` configuration module:

.. code-block:: python

    extensions = (...,
                  'numbadoc')

This extension adds two configuration fields, which determine the prefix
printed before jitted functions in the reference documentation.
Overwrite these values in your :file:`conf.py` to change this behaviour

.. code-block:: python

    #defaults
    numba_jit_prefix = '@numba.jit'
    numba_vectorize_prefix = '@numba.vectorize'
    numba_cuda_jit_prefix = '@numba.cuda.jit'

With the extension installed `autodoc` will automatically find
numba decorated objects and generate the correct docs.

If a vecotrized function with fixed signatures is found, these are injected
into the docstring.
"""
from typing import List, Iterator
from sphinx.domains.python import PyFunction
from sphinx.ext.autodoc import FunctionDocumenter

from numba.core.dispatcher import Dispatcher
from numba.np.ufunc.dufunc import DUFunc
from numba.cuda.compiler import AutoJitCUDAKernel

class NumbaFunctionDocumenter(FunctionDocumenter):
    """Document numba decorated functions."""

    def import_object(self) -> bool:
        """Import the object given by *self.modname* and *self.objpath* and set
        it as *self.object*.

        Returns True if successful, False if an error occurred.
        """
        success = super().import_object()
        if success:
            # Store away numba wrapper
            self.jitobj = self.object
            # And bend references to underlying python function
            if hasattr(self.object, "py_func"):
                self.object = self.object.py_func
            elif hasattr(self.object, "_dispatcher") and \
                 hasattr(self.object._dispatcher, "py_func"):
                self.object = self.object._dispatcher.py_func
            else:
                success = False
        return success

    def process_doc(self, docstrings: List[List[str]]) -> Iterator[str]:
        """Let the user process the docstrings before adding them."""
        # Essentially copied from FunctionDocumenter
        for docstringlines in docstrings:
            if self.env.app:
                # let extensions preprocess docstrings
                # need to manually set 'what' to FunctionDocumenter.objtype
                # to not confuse preprocessors like napoleon with an objtype
                # that they don't know
                self.env.app.emit('autodoc-process-docstring',
                                  FunctionDocumenter.objtype,
                                  self.fullname, self.object,
                                  self.options, docstringlines)
            # This block inserts information about precompiled signatures
            # if this is a precompiled vectorized function
            if getattr(self.jitobj, "types", []) and \
               getattr(self.jitobj, "_frozen", False):
                s = "| *Precompiled signatures:" + str(self.jitobj.types) + "*"
                docstringlines.insert(0, s)
            yield from docstringlines

class JitDocumenter(NumbaFunctionDocumenter):
    """Document jit/njit decorated functions."""

    objtype = 'jitfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return isinstance(member, Dispatcher) and hasattr(member, 'py_func')

class VectorizeDocumenter(NumbaFunctionDocumenter):
    """Document vectorize decorated functions."""

    objtype = 'vecfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return isinstance(member, DUFunc) and \
               hasattr(member, '_dispatcher') and \
               hasattr(member._dispatcher, 'py_func')

class CUDAJitDocumenter(NumbaFunctionDocumenter):
    """Document jit/njit decorated functions."""

    objtype = 'cudajitfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return isinstance(member, AutoJitCUDAKernel)\
               and hasattr(member, 'py_func')

class JitDirective(PyFunction):
    """Sphinx jitfun directive."""

    def get_signature_prefix(self, sig):
        return self.env.config.numba_jit_prefix

class VectorizeDirective(PyFunction):
    """Sphinx vecfun directive."""

    def get_signature_prefix(self, sig):
        return self.env.config.numba_vectorize_prefix

class CUDAJitDirective(PyFunction):
    """Sphinx jitfun directive."""

    def get_signature_prefix(self, sig):
        return self.env.config.numba_cuda_jit_prefix

def setup(app):
    """Setup Sphinx extension."""
    # Register the new documenters and directives (autojitfun, autovecfun)
    # Set the default prefix which is printed in front of the function signature
    app.setup_extension('sphinx.ext.autodoc')
    app.add_autodocumenter(JitDocumenter)
    app.add_directive_to_domain('py', 'jitfun', JitDirective)
    app.add_config_value('numba_jit_prefix', '@numba.jit', True)
    app.add_autodocumenter(VectorizeDocumenter)
    app.add_directive_to_domain('py', 'vecfun', VectorizeDirective)
    app.add_config_value('numba_vectorize_prefix', '@numba.vectorize', True)
    app.add_autodocumenter(CUDAJitDocumenter)
    app.add_directive_to_domain('py', 'cudajitfun', CUDAJitDirective)
    app.add_config_value('numba_cuda_jit_prefix', '@numba.cuda.jit', True)

    return {
        'parallel_read_safe': True
    }

[soleti]

Hey @HPLegion thank you very much for this. However, I get this error message

Could not import extension numbadoc (exception: cannot import name 'AutoJitCUDAKernel' from 'numba.cuda.simulator.compiler' (/usr/local/lib/python3.8/site-packages/numba/cuda/simulator/compiler.py))

which I think it's because I am running it on a machine without a GPU. Replacing it with FakeCUDAKernel fixes the error, but the functions still don't appear in the docs.

[HPLegion]

that's a curious one, cause my laptop does not have a GPU (and that is where I tested). Which version of numba are you running?

Does this work for you (or does it show a different type)?

In [1]: from numba.cuda import jit as cjit                                     

In [2]: @cjit 
   ...: def f(a, b): 
   ...:     return a+b 
   ...:                                                                        

In [3]: type(f)                                                                
Out[3]: numba.cuda.compiler.AutoJitCUDAKernel

[gmarkall]

AutoJitCUDAKernel was removed in 0.51 - it is now cuda.compiler.Dispatcher: https://github.com/numba/numba/blob/master/numba/cuda/compiler.py#L773 - this is part of bringing the CUDA target into line with the CPU target.

[HPLegion]

Sweet, that will likely be the reason (I am still running 0.50.0 locally) @soleti if you are running 0.51+ just exchange the AutoJitCUDAKernel to the cuda Dispatcher (both import and in CUDAJitDocumenter.can_document_member)

[soleti]

Uhm it still doesn't work. I am doing

from numba.cuda.compiler import Dispatcher as CUDADispatcher

which gives me

Extension error:
Could not import extension numbadoc (exception: cannot import name 'Dispatcher' from 'numba.cuda.simulator.compiler' (/usr/local/lib/python3.8/site-packages/numba/cuda/simulator/compiler.py))

[alexander-tritt-monash]

Hey! Thank you for the extension for autodoc. I wanted to add, for future people searching this, that this doesn't work with numba cuda device functions specifically, only kernels. To fix this, add this

from numba.cuda.compiler import DeviceFunctionTemplate

and change this

class CUDAJitDocumenter(NumbaFunctionDocumenter):
    """Document jit/njit decorated functions."""

    objtype = 'cudajitfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return isinstance(member, AutoJitCUDAKernel)\
               and hasattr(member, 'py_func')

to

class CUDAJitDocumenter(NumbaFunctionDocumenter):
    """Document jit/njit decorated functions."""

    objtype = 'cudajitfun'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        return (isinstance(member, AutoJitCUDAKernel) and hasattr(member, 'py_func')) or isinstance(member, DeviceFunctionTemplate)

Hope this helps someone, and thanks again!

[guilhermeleobas]

@HPLegion, thanks for the sphinx extension for autodoc. I tried to change it to work with autosummary + @intrinsic:

import numba
from typing import List, Optional
from sphinx.ext.autodoc import FunctionDocumenter

class NumbaDocumenter(FunctionDocumenter):
    """Sphinx directive that also understands numba decorators"""

    objtype = 'function'  # This must be assigned to 'function' to work!!
    directivetype = 'function'

    @classmethod
    def can_document_member(cls, member, membername, isattr, parent):
        if isinstance(member, numba.core.extending._Intrinsic):
            return True
        return super(RBCDocumenter, cls).can_document_member(member, membername, isattr, parent)

def setup(app):
    """Setup Sphinx extension."""

    # Register the new documenters and directives (autojitfun, autovecfun)
    # Set the default prefix which is printed in front of the function signature
    app.setup_extension('sphinx.ext.autodoc')
    app.add_autodocumenter(NumbaDocumenter)

    return {
        'parallel_read_safe': True
    }

Also, in order for this to work, one needs to update the intrinsic wrapper:

import functools
functools.update_wrapper(my_intrinsic, my_intrinsic._defn)  # _defn is the inner function

[gipert]

Hi @HPLegion, thanks for the extension! Any idea on how to extend it to support @guvectorize-decorated functions?

[gipert]

Hi @HPLegion, thanks for the extension! Any idea on how to extend it to support @guvectorize-decorated functions?

Here's my version of the extension that handles @guvectorized functions, if anyone's interested:

https://github.com/legend-exp/pygama/blob/f980faa5322aac1de12b5817b1d33c3248e399b5/docs/source/extensions/numbadoc.py

[HPLegion]

I am buried in other work at the moment, but with the collection as it is, this almost looks as if it might warrant releasing a micro-package that people can install or vendor directly into their codebase. If anyone feels the urge to create such a thing, I would certainly not hold a grudge :-P

Just had a brief look at the issue @gmarkall cross linked yesterday, seems like there are still issues linked to the missing magic wrapper information. Is there a dedicated issue for that?

[milannnnn]

Hi all. Found a simple way to document the Numba classes and methods in my code. Basically, I just added the following line to the Sphinx conf.py (effectively disabling the JIT compilation, and allowing Sphinx autodoc to properly parse all functions and classes):

os.environ["NUMBA_DISABLE_JIT"] = '1'

[SilvestriStefano]

Hello all, I am adjusting/improving/cleaning up the documentation (pr#160) for a python package. However, I have been struggling with documenting jit-decorated functions. I am using autosummary extension to generate recursively a page for every module/class/function (see SO#2701998 )

I have tried all the solutions mentioned here:

• Adding os.environ["NUMBA_DISABLE_JIT"] = '1' to the conf.py does nothing ;
• the autodoc extensions by @HPLegion, @gipert, @guilhermeleobas all return warnings and errors like the following

  
  WARNING: [autosummary] failed to import sarkas.potentials.
  Possible hints:

• AttributeError: module 'sarkas' has no attribute 'potentials'
• ImportError:
• TypeError: Failed in nopython mode pipeline (step: nopython frontend)

  
  and/or
  ```shell
  Failed to import sarkas.potentials.
  Possible hints:

• AttributeError: module 'sarkas' has no attribute 'potentials'
• TypeError: Failed in nopython mode pipeline (step: nopython frontend) issubclass() arg 2 must be a class or tuple of classes

  
  Somehow it can't read the submodules anymore?

I am using Sphinx v4.5.0, but I have also tried Sphinx v6.2.1 and v7.2.6 with the same results.

I don't really understand what I am doing wrong any help is highly appreciated.

[HPLegion]

@SilvestriStefano

Hi,

unfortunately, it has been a while since I have thought about this, so it is a bit tricky to give good advice. However: Those errors do not look like sphinx problems. It looks like there is actually a problem with the package you are documenting. If you run a pure Python shell in the environment that you build the documentation with, can you actually import that package as expected or do you get a similar error?

Cheers

[SilvestriStefano]

Hello @HPLegion , thanks for responding. The issue only arises whenever I add the extra extension in the conf.py. Otherwise the documentation builds fine without warnings, except that the module functions are not present (see below)

The conf.py with the extension looks like :

import os
import sys

import sarkas

sys.path.insert(0, os.path.abspath("../sarkas"))
sys.path.insert(0, os.path.abspath("../sarkas/time_evolution"))
sys.path.insert(0, os.path.abspath("../sarkas/utilities"))
sys.path.insert(0, os.path.abspath("../sarkas/potentials"))
sys.path.insert(0, os.path.abspath("../sarkas/tools"))
sys.path.insert(0, os.path.abspath("./_ext"))        # <-------- this is where the extension file is located

extensions = [
    "sphinx.ext.napoleon",  
    "sphinx.ext.autodoc",  
    "sphinx.ext.autosummary", 
    "numbadecoratordoc"                  # <-------- how i called the file
]

# ... more stuff ...

autodoc_mock_imports = ["yaml", "numba", "scipy", "optparse", "time", "pyfftw", "pyfiglet", "tqdm", "fmm3dpy"]

# Generate the API documentation when building
autosummary_generate = True
autosummary_generate_overwrite = True
# # A list of ignored prefixes for module index sorting.
modindex_common_prefix = ["sarkas."]

# Make sure the target is unique
autosectionlabel_prefix_document = True

# Napoleon settings
napoleon_google_docstring = False
napoleon_numpy_docstring = True
napoleon_include_init_with_doc = False
napoleon_include_private_with_doc = False
napoleon_include_special_with_doc = True
napoleon_use_admonition_for_examples = False
napoleon_use_admonition_for_notes = False
napoleon_use_admonition_for_references = False
napoleon_use_ivar = True
napoleon_use_param = False
napoleon_use_rtype = False

[SilvestriStefano]

Hello again @HPLegion , I have done a little debugging and i found out a few things:

• the warnings I mentioned above are because in some instances we add signature specifications to the decorator. Sometimes (see the create_cells_array() in the screen of my previous comment) they can create more problems like

  
  Untyped global name 'zeros': Cannot determine Numba type of <class 'builtin_function_or_method'>

File "../sarkas/potentials/force_pp.py", line 538: def create_cells_array(box_lengths, cutoff):

cells_per_dim = zeros(3, dtype=int64)
^

 It is not a big problem (to us anyway) as apparently we can go without the signature specifications.
- if I only use the decorators `@jit(nopython=True)` or `@njit` I get no errors and no warnings, however the decorated functions are still not appearing in the doc.
- If I modify the `JitDocumenter` class method `can_document_member` as follows 
```python
@classmethod
def can_document_member(cls, member, membername, isattr, parent):
    return isinstance(member, Dispatcher) and hasattr(member, 'py_func') and member.__doc__ is None

the decorated functions (they should not have any signature specification) whose docstring is not empty do appear in the docs but without the prefix. I am confused as why this works though.

• If I add the file .numba_config.yaml with the entry DISABLE_JIT: "1" in the docs/ folder (as per numba documentation) and use your extension without modifications, all the decorated functions (even those without a docstring and those with signature specifications) appear in the documentation.
• If I add the file .numba_config.yaml with the entry DISABLE_JIT: "1" in the docs/ folder and do not use your extension, no decorated functions appears in the documentation.

[HPLegion]

Hi Stefano (and others)

I just wanted to give you a heads-up that I won't be able to follow up on these issues myself for a while, unfortunately. I would like to offer more help, but time is limited at the moment.

I expect documentation to fail without the plugin (as you observed), since Sphinx gets confused by numba's decorators. I don't know why you see the flaky behviour in some cases though.

When I first looked into this, I really had to debug the Sphinx autodoc run to see why the documenters did not pick up on the decorated functions, and then I tried to come up with something that would be able to identify the decorated functions as more or less normally documentable objects. Then I used the chance to sprinkle in a few extras, maybe some of that broke over the last years. It is not particularly fun, but maybe a way for follow up investigations.

In the meantime I would encourage you to write on https://numba.discourse.group/ - there is a chance you will be able to get in touch with some knowledgable people who are not aware of this issue / disucussion. Judging by how lively this thread still is, this seems to affect a number of users. Maybe someone has the time and energy to really dive into this and come up with a good and maintainable lonterm solution :-)

Best of luck!

[SilvestriStefano]

At the moment we are satisfied with how it works. However, I would prefer to understand the flaky behavior, so I will probably bug the people on discourse :P

Thank you for all your help :)