search

jdillard / sphinx-sitemap

Sphinx extension to generate a multi-lingual, multi-version sitemap for HTML builds
https://sphinx-sitemap.readthedocs.io/en/latest/index.html
MIT License
55 stars 22 forks source link

sphinx-sitemap >v2.4.0 raises execption during sitemap creation. #96

Open nzlosh opened 2 days ago

nzlosh commented 2 days ago

Linux Distribution

echo $(source /etc/os-release; echo $VERSION)
22.04.5 LTS (Jammy Jellyfish)

Python Version

python3 --version
Python 3.10.12

Sphinx Version

sphinx-build --version
sphinx-build 8.1.3

Problem description

Versions of sphinx-sitemap priori to v2.4.0 generate the sitemap successfully as shown here with v2.3.0.

[app] emitting event: 'build-finished'(None,)
/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx_sitemap/__init__.py:166: RemovedInSphinx90Warning: Sphinx 9 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
  filename = app.outdir + "/" + app.config.sitemap_filename
sphinx-sitemap: sitemap.xml was generated for URL https://docs.stackstorm.com/ in /root/workspace/st2docs/docs/build/html/sitemap.xml
build succeeded.

The HTML pages are in docs/build/html.

Build finished. The HTML pages are in docs/build/html.

All versions v2.4.0 and greater fail with the following exception

 Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/events.py", line 404, in emit
    results.append(listener.handler(self.app, *args))
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx_sitemap/__init__.py", line 111, in add_html_link
    env.sitemap_links.put(sitemap_link)
  File "<string>", line 2, in put
  File "/usr/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/usr/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 508, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
BlockingIOError: [Errno 11] Resource temporarily unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/application.py", line 381, in build
    self.builder.build_update()
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 358, in build_update
    self.build(
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 437, in build
    self.write(docnames, list(updated_docnames), method)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 711, in write
    self.write_documents(docnames)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 725, in write_documents
    self._write_serial(sorted_docnames)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 744, in _write_serial
    self.write_doc(docname, doctree)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 659, in write_doc
    self.handle_page(docname, ctx, event_arg=doctree)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 1162, in handle_page
    newtmpl = self.app.emit_firstresult(
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/application.py", line 836, in emit_firstresult
    return self.events.emit_firstresult(event, *args,
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/events.py", line 433, in emit_firstresult
    for result in self.emit(name, *args, allowed_exceptions=allowed_exceptions):
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/events.py", line 415, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <function add_html_link at 0x7fda6660b010> for event 'html-page-context' threw an exception (exception: [Errno 11] Resource temporarily unavailable)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/events.py", line 404, in emit
    results.append(listener.handler(self.app, *args))
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx_sitemap/__init__.py", line 128, in create_sitemap
    if env.sitemap_links.empty():
  File "<string>", line 2, in empty
  File "/usr/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/usr/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 508, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
BlockingIOError: [Errno 11] Resource temporarily unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/cmd/build.py", line 514, in build_main
    app.build(args.force_all, args.filenames)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/application.py", line 389, in build
    self.events.emit('build-finished', err)
  File "/root/workspace/st2docs/virtualenv/lib/python3.10/site-packages/sphinx/events.py", line 415, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <function create_sitemap at 0x7fda6660b0a0> for event 'build-finished' threw an exception (exception: [Errno 11] Resource temporarily unavailable)

Extension error (sphinx_sitemap):
Handler <function create_sitemap at 0x7fda6660b0a0> for event 'build-finished' threw an exception (exception: [Errno 11] Resource temporarily unavailable)
make: *** [Makefile:67: .docs] Error 2

No changes other than upgrading from 2.3.0 to 2.4.0 were made. Any assistance on how to resolve this issue would be most appreciated.

jdillard commented 1 day ago

Ah interesting, looks like you have a race condition of sorts, I haven't seen that yet. Was it reproducible, or did it only happen once?

Looks like I could try using multiprocessing.Queue instead of Manager().Queue() and also add some better error handling, but regardless it would take a patch release to fix. In the mean time you can pin the extension or try running in serial mode for now (2.4.0 added parallel support).

nzlosh commented 6 hours ago

Yes, it's reproducible, it occurs on every execution. It appears the documentation generation process we're using pulls in eventlet (I didn't know this). After import, the below message is emitted:

1 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.

I suspect this is the cause of the exception so I'll try to investigate this further.

nzlosh commented 6 hours ago

Possibly related issues https://github.com/gevent/gevent/issues/1808 and/or https://github.com/gevent/gevent/issues/1268