picnixz
commented
10 months ago The issue is likely because of circular references since you are sharing everything with everyone. And you likely have references to things that should not.
I have two ideas:
- verify your references and only include whatever needs to be included; don't share more what should be needed. I think your environment has too much circular references. So, before implementing a solution on our side, check that your objects do not have circular references, and if they do have, I would suggest coming up with an alternative.
-
we protect
is_serializableagainst self-recursions by detecting whether we are comparing things that were already being compared. However, we cannot protect against this:d = {'a': {'b': 2}} d['a'] = dWith that structure, there is no way we can actually check 'b' (and actually you'll never be able to get the value of 'b' itself so it's not really an issue).
For now, I'd say it's better you stick with an older Sphinx version but you should carefully decide whether you need circular references or not (but since it worked for the past years you'll likely not change). I can come up with a fix for handling non-pathological cases because it's just using the same idea as reprlib.recursive_repr.
Describe the bug
Sphinx Version: 7.1.2
We are running into a RecursionError that impedes Sphinx's ability to process documentation content and generate output. That error is encountered during execution of the is_serializable function defined in sphinx/config.py.
Given that our documentation suite is very large and that each of our documents can be hundreds of pages long, we require flexibility in defining our documents' contents. We employ a combination of .yml (key:value pairs) and .j2 (content templates) files to build the content that is fed to Sphinx for pickling and generating output.
Since we support several different system configurations, we must vary the number of and data for each of the Jinja contexts we define for building the content that is fed to Sphinx. In rudimentary terms, a main context is composed of its core block of data plus two lists: a) contexts_list which contains the data from sibling contexts, and b) additional_contexts_list which contains the data from children contexts. We came up with that scheme to allow all contexts to be visible at the same time and be able to collect data from all of them when building a document. We have been using this approach for the past 6 years while using older versions of Sphinx.
It is when encountering either our contexts_list or additional_contexts_list that the RecursionError is produced. Forcing Sphinx 7.1.2 to bypass the is_serializable function or running the exact same data with an older Sphhinx version that does not have that check/function allows Sphinx to successfully generate the .tex and .pdf files we are after.
Each sibling or child context block of data can be estimated at below 10MB and above 1MB.
Can is_serializable be modified to handle the approach from above? ALTERNATIVELY, does Sphinx provide a way to access data from another context when working within one context?
How to Reproduce
I cannot copy-paste the details or amount of technical data that makes up each environment within either the contexts_list or the additional_contexts_list in here. The proprietary nature and volume of the data tie my hands.
As questions come up, I can work with someone to answer them.
The high-level command we issue is "sphinx-build -b latex -d build/doctrees source build/latex".
Environment Information
Sphinx extensions
Additional context
No response