search

jdillard / sphinx-sitemap

Sphinx extension to generate a multi-lingual, multi-version sitemap for HTML builds
https://sphinx-sitemap.readthedocs.io/en/latest/index.html
MIT License
55 stars 21 forks source link

Doesn't reference pages in subdirectories #20

Open samccann opened 5 years ago

samccann commented 5 years ago

I tried running this against the Ansible documentation, which includes multiple subdirectories, and I only get results from the parent directory.

Generated sitemap

https://docs.ansible.com/latest/404.html https://docs.ansible.com/latest/genindex.html https://docs.ansible.com/latest/py-modindex.html https://docs.ansible.com/latest/search.html https://docs.ansible.com/latest/opensearch.html

Details

jdillard commented 5 years ago

It looks like this is due to sphinx running in multiprocessor mode. When I remove -j 4 from the build command, it seems to include all the pages.

I think the fix is to figure out how to move the sitemap_links list into shared memory. I haven't figured out how to do that just yet, but it probably looks something like sphinx.ext.todo.

Once that is working, the following can be added to setup:

return {
    'parallel_read_safe': True
}
samccann commented 5 years ago

Tried that locally (removing -j ) and it does seem to have all the entries in there now.

jdillard commented 5 years ago

Great. It looks like it should be switching to a serial read, because it throws these warnings in the beginning: 

WARNING: the sphinx_sitemap extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit
WARNING: doing serial read

But maybe it isn't switching properly and there is a bug somewhere preventing that. I'll keep looking, but it's not looking like a small change.

samccann commented 5 years ago

no worries. Thanks for digging into this. I at least have a way to create it when I need to.

toshiakiasakura commented 2 years ago

I was also stuck in this problem, when I developed my own OSS. I used "nbsphinx" to generate a site. At first, I thought this was source of a problem, but I found this issue and I suspected that subdirectories might be a cause.

Also, I could not figure out the above discussion. How to remove -j 4, since I always type sphinx-build ./docs_src ./docs to create docs, there is no -j 4 for me... Also, if setting return { 'parallel_read_safe': True} solves this issue, I also can not figure out where to set it. I just tried to set the file instructed in sphinx.ext.todo, but the problem was not solved...

Details