[Bug] Published addons manual pages not removing deleted plugins, TOC not updated

[echoix]

Name of the addon Let's talk about the recently removed g.cloud plugin from the grass8 branch.

Describe the bug

675 removed the g.cloud plugin from the grass8 branch. It is however still available in the table of contents of https://grass.osgeo.org/grass80/manuals/addons/, and the page https://grass.osgeo.org/grass80/manuals/addons/g.cloud.html is still online

To Reproduce Steps to reproduce the behavior:

1. Go to https://grass.osgeo.org/grass80/manuals/addons/
2. Find the g.cloud addon.
3. Click on g.cloud (https://grass.osgeo.org/grass80/manuals/addons/g.cloud.html)
4. Go to the bottom of the page, see that there isn't the Last-changed note that was added recently in the mkhtml.py update. (Meaning it wasn't built recently)

Expected behavior When a plugin is removed, the table of contents is updated, and the page is removed too. If a plugin is renamed, the old name should be removed, and keeping the new page online.

Screenshots https://grass.osgeo.org/grass80/manuals/addons/ Table of contents: The g.cloud page: This page wasn't built recently:

System description (please complete the following information):

• Operating System: Windows 11 21H2
• GRASS GIS version 8.0
• Chrome Version 97.0.4692.71 (Official Build) (64-bit)

Additional context It seems that there is no deletion of files, source files are cached between documentation builds.

[neteler]

Thanks for catching that.

I checked on the server:

neteler@grasslxd:~$ cd src/grass8-addons/src/general/
✔ ~/src/grass8-addons/src/general [grass8|✔]  $ ls
g.citation/     g.download.location/  g.proj.identify/  Makefile
g.compare.md5/  g.isis3mt/            g.projpicker/
g.copyall/      g.proj.all/           g.rename.many/

The module has been properly removed from the addons tree.

However, it has not been removed from the manual directory:

neteler@grasslxd:/var/www/code_and_data/grass80/manuals/addons/$ ls -la g.cloud.html 
-rw-rw-r-- 1 neteler users 9253 Jan 13 05:34 g.cloud.html

I suspect that the cronjob lacks a proper cleaning of this directory before repopulating it with the latest HTML pages.

Importantly, the time between deleting the old pages and repopulating the directory should be close to 0 sec in order to avoid that visitors in that moment see 404 errors in the manual. Means: the fresh compilation results need to be staged temporarily and then replace the now outdated addons/ subdirectory in one step. Hence, this section needs a review:

https://github.com/OSGeo/grass-addons/blob/df7009f02d1c4431e6c8f81a9d22c7d99b9b461f/utils/cronjobs_osgeo_lxd/cron_grass8_relbranch_build_binaries.sh#L263

[echoix]

I see two kind of approaches. One where we are able to have all the new folder structure done besides in another folder, and like "rename" the "addons" folders to the new one, but I imagine undesirable edge cases if it fails. From trying to build the man locally, I saw that the css was only generated by the core repo build. So only running the build in a secondary folder might not be enough. Is the addons manual pages built from scratch each time, or make is able to skip unchanged files? Could an rsync like approach be used to compare two subfolders (with delete) once validated that the build is successful? Or, since deletes or renames are rare, just a checker job run weekly could patch the problem too.

Could editing the nginx configuration to redirect to a new folder be safe?

[neteler]

Is the addons manual pages built from scratch each time

Yes, unless the aforementioned cronjob script is significantly changed (if possible at all).

rsync with properly chosen flags might be an option.

(it is an Apache server - your comment about the config isn't yet clear to me).