Exponential Increase in Build Time with Multiple :numref: References and Custom-Extension Tables

roei-bloch commented 3 months ago

Describe the bug

I created a Sphinx extension to handle grid tables, adding some functionality. To allow future use of the :numref: role to reference these tables, I used the add_name(self, node) function in the extension code to register the table as an explicit target. However, the build time increases exponentially as more tables and numrefs references them are added, becoming unmanageable very quickly.

I examined the issue, and the results indicate that the problem lies in the code resolving the :numref: references for tables registered as targets using the add_name function.

Results:

Y-axis: Build time X-axis: Number of tables and :numref: references in the files

Graph 1: Build time for custom-extension tables referenced by :numref:. The build time increases exponentially, becoming unmanageable after surpassing 600 tables. graph1

Graph 2: Comparison between custom-extension tables and regular tables build time, both referenced by :numref:.

Graph 3: Comparison between custom-extension tables with and without :numref: references, proving that the issue lies in the code resolving the :numref: references.

How to Reproduce

extention code:

from docutils import nodes
from HTMLtoRST.tables_converter import *
from docutils.parsers.rst import Directive, directives
from docutils.statemachine import StringList
from docutils.statemachine import ViewList
from docutils.parsers.rst.tableparser import TableParser

class adbtable(Directive):
    required_arguments = 1
    optional_arguments = 2
    final_argument_whitespace = True
    option_spec = {
        'title': directives.unchanged,
        'adbname': directives.unchanged,
    }
    has_content = True

    def run(self):
        title = (' '.join(self.arguments))
        adbname = self.options.get('adbname', '')
        self.options['name'] = adbname
        par_node = nodes.paragraph(ids=[nodes.make_id(adbname)])

        desc_table_node = nodes.table()
        desc_table_caption = nodes.caption('', title)
        desc_table_node += desc_table_caption
        self.state.nested_parse(self.content, self.content_offset, desc_table_node)
        self.add_name(desc_table_node) # make adbname the anchor refered by numref (:numref:`adbname`)
        par_node += desc_table_node

        # self.add_name(section_node['ids'][0]) #for numref
        return [desc_table_node]

def setup(app):
    app.add_directive('adbtable', adbtable)

after adding this extension and declare its existence in conf.py, create a rst file add use this new directive, and use numref to reference it. the many tables and reference you add build time will increase exponentially. here is a python script that can generate tables and numrefs for you: (modify the amount of tables and numrefs as you wish)

print("################\noutput\n################\n\n")

for i in range(100):
    print(f":numref:`Table{i}`")

print("\n")

for i in range(100):
    print(f".. adbtable:: table_name\n   :adbname: Table{i}\n\n   +---------+-------+----------+-------------+--------+\n   | Offset  | Bits  | Name     | Description | Access |\n   +=========+=======+==========+=============+========+\n   | 000     | 00000 | output   |             |        |\n   +---------+-------+----------+-------------+--------+")

Environment Information

Platform:              linux; (Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)
Python version:        3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        7.2.5
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

Sphinx extensions

No response

Additional context

No response

AA-Turner commented 3 months ago

Environment Information

Platform:              linux; (Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)
Python version:        3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        7.2.5
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

Hi Roei (@roei-bloch), the versions of Sphinx and Docutils you're using are fairly old. Please could you test with Python 3.12, Sphinx 7.4, and Docutils 0.21?

A

roei-bloch commented 3 months ago

Thanks for the quick answer! I will try.

Thanks, Roei

From: Adam Turner @.> Sent: Wednesday, July 17, 2024 5:32:29 PM To: sphinx-doc/sphinx @.> Cc: Roei Bloch @.>; Mention @.> Subject: Re: [sphinx-doc/sphinx] efficiency issue: build time increase exponential when number of :numref: and tables occurrences grows (Issue #12611)

Environment Information

Platform: linux; (Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35) Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]) Python implementation: CPython Sphinx version: 7.2.5 Docutils version: 0.20.1 Jinja2 version: 3.1.2 Pygments version: 2.16.1

Hi Roei @.***https://github.com/roei-bloch), the versions of Sphinx and Docutils you're using are fairly old. Please could you test with Python 3.12, Sphinx 7.4, and Docutils 0.21?

A

— Reply to this email directly, view it on GitHubhttps://github.com/sphinx-doc/sphinx/issues/12611#issuecomment-2233474561, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYPIQUTVDFF54X5OSVPDFH3ZMZ573AVCNFSM6AAAAABLAYEGA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTGQ3TINJWGE. You are receiving this because you were mentioned.Message ID: @.***>

External e-mail, be judicious when opening attachments or links

sphinx-doc / sphinx