search

cschwan / sage-on-gentoo

(Unofficial) Gentoo Overlay for Sage- and Sage-related ebuilds
84 stars 26 forks source link

pickling issues building sage-doc-9999 with use=doc-pdf and all available LANGS #681

Closed strogdon closed 3 weeks ago

strogdon commented 2 years ago

The most pressing situation seems to be on Prefix although I occasionally have issues on Gentoo. The pickling issues are similar to https://github.com/cschwan/sage-on-gentoo/issues/667#issuecomment-1004356911 when building the standard sage-doc-9999. Doing 

emake doc-html PYTHON=${PYTHON}

seems to work and

emake doc-pdf PYTHON=${PYTHON}

also appears to work. This latter case appears to build some html stuff under html/en. The following

emake doc-html PYTHON=${PYTHON} && emake doc-pdf PYTHON=${PYTHON}

fails with

MAKE=make /storage/strogdon/gentoo-rap/usr/bin/python3.10 -m sage_docbuild reference/spkg pdf --no-prune-empty-dirs
[tensor_fr] dumping search index in English (code: en)... done
[tensor_fr] The HTML pages are in ../../../build_doc/html/en/reference/tensor_free_modules.
Build finished. The built documents can be found in /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-doc-9999/work/build_doc/html/en/reference/tensor_free_modules
MAKE=make /storage/strogdon/gentoo-rap/usr/bin/python3.10 -m sage_docbuild reference/manifolds pdf --no-prune-empty-dirs
Error building the documentation.
Traceback (most recent call last):
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 1745, in main
    builder()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 247, in pdf
    self.latex()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 776, in _wrapper
    for module_name in self.get_new_and_updated_modules():
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 946, in get_new_and_updated_modules
    env = self.get_sphinx_environment()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 843, in get_sphinx_environment
    env = pickle.load(f)
_pickle.UnpicklingError: pickle data was truncated

    Note: incremental documentation builds sometimes cause spurious
    error messages. To be certain that these are real errors, run
    "make doc-clean doc-uninstall" first and try again.
[repl     ] dumping search index in English (code: en)... done
[repl     ] The HTML pages are in ../../../build_doc/html/en/reference/repl.
Build finished. The built documents can be found in /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-doc-9999/work/build_doc/html/en/reference/repl

The point where the failure occurs is not predictable. So far I have been unable to build html + pdf docs for 9.6.beta3 on Prefix.

kiwifb commented 2 years ago

I had to build at -j6 instead of -j8 with just two languages. But I squarely got a https://github.com/cschwan/sage-on-gentoo/issues/667#issuecomment-1003882877 kind of issue. The case that give errors would be the standard vanilla procedure. Default make produces html doc and then you have to do make doc-pdf to get the pdf, although there is probably a way to request it from the top.

I am assuming that what I put in which should be make doc-html doc-pdf doesn't work either?

strogdon commented 2 years ago

Yes, when I do the make doc-html doc-pdf I get the https://github.com/cschwan/sage-on-gentoo/issues/667#issuecomment-1003882877 kind of issue. Not sure why the make doc-html and then make doc-pdf fails. I googled the pickle error above and it may appear when a pickle is empty.

kiwifb commented 2 years ago

It is just another kind of parallel make issue where a task starts before the necessary previous one has ended. But where to tweak?

strogdon commented 2 years ago

It appears that when one does make doc-html and then make doc-pdf that some html stuff (html/en) is rebuilt. I don't think this should be the case. Shouldn't the make process know the html/en is up to date and thus not rebuild it. Maybe they are different make processes. FWIW I do have an 8-core machine with MAKEOPTS="-j9" where sage-doc-9999 built just fine.

kiwifb commented 2 years ago

Indeed some bits of html are rebuilt and in an ideal world they shouldn't. make does look for files as targets, when they are more abstract like it is the case here you can have those kind of issues. In fact in an ideal world running make doc-html would just check a number of files are present and do nothing. This is not the case at present. If we had a clear file as a final target we could avoid that kind of re-build.

strogdon commented 2 years ago

The host of the prefix has 8 cores. I'm only able to get sage-doc-9999 to build serially, -j1. It's very tedious.

kiwifb commented 2 years ago

Sounds seriously tough. I really need to bring back pre-build doc in stable if we cannot fix or mitigate that.

kiwifb commented 2 years ago

Latest merge on the vbraun branch from 6 hours ago, doesn't have anything about sage_docbuild or sage-doc, and it just build the doc like there was no problems. Unlike the previous merge which would have been identical to 9.6.beta3.

strogdon commented 2 years ago

Some thoughts. When USE=doc-pdf, in src/doc/Makefile remove respectively, doc-inventory-reference and doc-html-reference from the targets doc-pdf-reference and doc-pdf-other and build the docs as 

emake doc-html PYTHON=${PYTHON} && emake doc-pdf PYTHON=${PYTHON}

then each component html and pdf builds in parallel - html first and pdf next. Not strictly in parallel but it does build on the Prefix.

kiwifb commented 2 years ago

Got hit on the latest vbraun merge. '-j8' unknown character, '-j6' 'FileNotFoundError'. So, I am going to try what you are suggesting in your latest message to see if things improve.

kiwifb commented 2 years ago

571d34bd65d30ccd951df351fa399f42598d6a50 helped. At a cost of 2 minutes of build time (15mn instead of 13) probably a bargain. It is not exactly pleasant in my opinion but it may have to do for now.

strogdon commented 2 years ago

One machine where sage-doc-9999 from the overlay built. Two machines, one the subject Prefix, needed the fix to build.