search

jbms / sphinx-immaterial

Adaptation of the popular mkdocs-material material design theme to the sphinx documentation system
https://jbms.github.io/sphinx-immaterial/
Other
177 stars 28 forks source link

Update for sphinx v7.3.x #346

Closed 2bndy5 closed 3 weeks ago

2bndy5 commented 2 months ago

resolves #345

2bndy5 commented 2 months ago

I think there's a regression about consecutive builds using cpp.apigen. I now get warnings about documents not included in any toctree upon rebuilding docs (without flushing old build artifacts):

C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array-ce8a1651-06f1923e.rst: WARNING: document isn't included in any toctree
C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.DataOrder-62e6a0a2-a428c0b6.rst: WARNING: document isn't included in any toctree
C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.IndexInterval-562a996b-19da0973.rst: WARNING: document isn't included in any toctree

...

C:\dev\sphinx-immaterial\sphinx_immaterial.apidoc.cpp.apigen:1: WARNING: toctree contains reference to nonexisting document 'cpp_apigen_generated/cpp_apigen_demo.IndexInterval-562a996b'
C:\dev\sphinx-immaterial\sphinx_immaterial.apidoc.cpp.apigen:1: WARNING: toctree contains reference to nonexisting document 'cpp_apigen_generated/cpp_apigen_demo.DataOrder-62e6a0a2'
C:\dev\sphinx-immaterial\sphinx_immaterial.apidoc.cpp.apigen:1: WARNING: toctree contains reference to nonexisting document 'cpp_apigen_generated/cpp_apigen_demo.Array-ce8a1651'

I think there's an undesired behavior specific to Windows (or any Case sensitive file system) in which a consecutive build fails after a certain number of rebuilds. It seems that a hash is added to the file name each time a build is invoked, and at some point the path to the generated file becomes too long. This happens with both python apigen and cpp apigen.

jbms commented 2 months ago

We may need to investigate exactly how the previous state is reloaded when sphinx does an incremental build.

cpp apigen stuffs some extra attributes into app.env, and then define a _builder_inited that modifies it in place to add the hashes.

I think we may similarly add attributes to app.env elsewhere in this theme, and I think in all cases we don't want that data to be persisted to the next build.

Therefore we may need to change where it is stored --- possibly on the app object itself would be better.

2bndy5 commented 2 months ago

I think I found the problem. https://github.com/jbms/sphinx-immaterial/blob/61c3ef0fc8361f1e7e79c5953abd4d23ca52d1b7/sphinx_immaterial/apidoc/cpp/apigen.py#L897-L902 entities["page_name"] gets appended another hash every time _builder_inited() runs.

I think the fix is to consolidate the for loop in above link with the for loop located just afterward in the locally-scoped function https://github.com/jbms/sphinx-immaterial/blob/61c3ef0fc8361f1e7e79c5953abd4d23ca52d1b7/sphinx_immaterial/apidoc/cpp/apigen.py#L903-L915 My idea is to calc the hash (which based on os.path.basename(entities["pagename"])) without caching the hash-appended file name. Honestly though, I think hashing the bytes written to the file might be a bit more of a cache buster.

2bndy5 commented 2 months ago

Well I was able to use the file content (in bytes) as the hash seed, and it did prevent writing new files when the content hasn't changed (for both python and cpp apigen). See https://github.com/jbms/sphinx-immaterial/pull/346/commits/f97200f58ce513498d7d029407245614c85abad1

Now, I'm noticing that the site index is generated for 99 pages on a fresh doc build, but the site index is only generated for 63 files on a subsequent build.

jbms commented 2 months ago

Well I was able to use the file content (in bytes) as the hash seed, and it did prevent writing new files when the content hasn't changed (for both python and cpp apigen). See f97200f

Now, I'm noticing that the site index is generated for 99 pages on a fresh doc build, but the site index is only generated for 63 files on a subsequent build.

We don't want to use the file content because in fact the purpose is not cache busting but the opposite --- to ensure a stable URL even as the content changes when the documentation updates. The hash is to deal with case insensitive filesystems where we may have two document names that differ only in casing.

I think there may have been a change in how incremental rebuild state is saved. We want to make sure that this information is not saved across rebuilds, so I think we should move it out of app.env if app.env is getting pickled.

2bndy5 commented 1 month ago

We want to make sure that this information is not saved across rebuilds, so I think we should move it out of app.env if app.env is getting pickled.

I did this, but now I think we've hit another problem.

Builds with Windows now emit these warnings

> checking consistency... C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array-ce8a1651.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array.Array-convert-338c580d.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array.Array-data-shape-ba8897f2.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array.data-b9e4202e.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array.operator-subscript-257a2cc4.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array.order-4d0a1cd0.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Array.shape-c9344e8b.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.DataOrder-62e6a0a2.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.IndexInterval-562a996b.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.IndexInterval.IndexInterval-46626a6e.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.IndexInterval.lower-024c21ab.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.IndexInterval.operator-shift_left-b727dea4.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.IndexInterval.upper-aa98da78.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.Union-a8b425ad.rst: WARNING: document isn't included in any toctree C:\dev\sphinx-immaterial\docs\cpp_apigen_generated/cpp_apigen_demo.operator-shift_left-6191f4da.rst: WARNING: document isn't included in any toctree done

along with these related warnings

> C:\dev\sphinx-immaterial\sphinx_immaterial.apidoc.cpp.apigen:1: WARNING: toctree contains reference to nonexisting document 'cpp_apigen_generated/cpp_apigen_demo.IndexInterval' C:\dev\sphinx-immaterial\sphinx_immaterial.apidoc.cpp.apigen:1: WARNING: toctree contains reference to nonexisting document 'cpp_apigen_generated/cpp_apigen_demo.DataOrder' C:\dev\sphinx-immaterial\sphinx_immaterial.apidoc.cpp.apigen:1: WARNING: toctree contains reference to nonexisting document 'cpp_apigen_generated/cpp_apigen_demo.Array' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:11: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.IndexInterval' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:11: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.IndexInterval' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:11: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.DataOrder' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:29: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:36: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.Array-data-shape' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:44: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.Array-convert' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:47: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.data' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:50: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.shape' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:53: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.order' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:59: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.operator-subscript' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:59: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Array.operator-subscript' C:\dev\sphinx-immaterial\cpp_apigen_demo\array.h:21: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.operator-shift_left' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:17: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.IndexInterval.IndexInterval' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:20: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.IndexInterval.operator-shift_left' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:25: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.IndexInterval.lower' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:28: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.IndexInterval.upper' C:\dev\sphinx-immaterial\cpp_apigen_demo\index_interval.h:34: WARNING: unknown document: '/cpp_apigen_generated/cpp_apigen_demo.Union'

2bndy5 commented 1 month ago

I'm not sure why, but libclang (v18.1.1) on macos is failing to parse the cpp.apigen demo sources. It seems related to sighingnow/libclang#71

I don't have a mac I could use to test/debug this one.

jbms commented 3 weeks ago

I'm not sure why, but libclang (v18.1.1) on macos is failing to parse the cpp.apigen demo sources. It seems related to sighingnow/libclang#71

I don't have a mac I could use to test/debug this one.

Perhaps we can just add a libclang<17 requirement as a workaround until this gets fixed upstream?