HTML pages fail to build when parallel building with CMake is enabled

[mojca]

When trying to build documentation (ZZIP_HTMLSITE, ZZIP_HTMPAGES both set to `ON') with parallel build, the builds fail, for example with:

FileNotFoundError: [Errno 2] No such file or directory: 'zzipfseeko.xml'

by forcing a non-parallel build, everything seems to work just fine.

(Tested on macOS 11 arm64 using CMake-based build.)

[gdraheim]

Would you like to provide the full config line (cmake call) to show how to reproduce the case?

[mojca]

I can reproduce it with the following two lines (I need to disable the tests, else the build doesn't work at all, but that's for another ticket):

cmake -DZZIPSDL=OFF -DZZIPTEST=OFF -DZZIPDOCS=ON -DZZIP_HTMLSITE=ON -DZZIP_HTMPAGES=ON /path/to/gdraheim-zziplib-6b6242e
/usr/bin/make -j8 -w all VERBOSE=ON

The full invocation is below, just for the reference, but there's probably nothing relevant in there:

/opt/local/bin/cmake -G "CodeBlocks - Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/opt/local" -DCMAKE_INSTALL_NAME_DIR="/opt/local/lib" -DCMAKE_SYSTEM_PREFIX_PATH="/opt/local;/usr" -DCMAKE_C_COMPILER="/usr/bin/clang" -DCMAKE_CXX_COMPILER="/usr/bin/clang++" -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_POLICY_DEFAULT_CMP0060=NEW -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_COLOR_MAKEFILE=ON -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_MAKE_PROGRAM=/usr/bin/make -DCMAKE_MODULE_PATH="/opt/local/share/cmake/Modules" -DCMAKE_PREFIX_PATH="/opt/local/share/cmake/Modules" -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DCMAKE_INSTALL_RPATH="/opt/local/lib" -Wno-dev -DZZIPSDL=OFF -DZZIPTEST=OFF -DZZIPDOCS=ON -DZZIP_HTMLSITE=ON -DZZIP_HTMPAGES=ON -DPYTHON_EXECUTABLE:FILEPATH=/opt/local/bin/python3.9 -DCMAKE_OSX_ARCHITECTURES="arm64" -DCMAKE_OSX_DEPLOYMENT_TARGET="11.2" -DCMAKE_OSX_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" /path/to/zziplib-0.13.72
/usr/bin/make -j8 -w all VERBOSE=ON

Here's some sample output:

[ 95%] Generating zzipmmapped.xml
cd /tmp/build-libzzip/docs && /usr/bin/python3 /path/to/gdraheim-zziplib-6b6242e/docs/makedocs.py /path/to/gdraheim-zziplib-6b6242e/zzip/dir.c /path/to/gdraheim-zziplib-6b6242e/zzip/err.c /path/to/gdraheim-zziplib-6b6242e/zzip/fetch.c /path/to/gdraheim-zziplib-6b6242e/zzip/file.c /path/to/gdraheim-zziplib-6b6242e/zzip/fseeko.c /path/to/gdraheim-zziplib-6b6242e/zzip/info.c /path/to/gdraheim-zziplib-6b6242e/zzip/memdisk.c /path/to/gdraheim-zziplib-6b6242e/zzip/mmapped.c /path/to/gdraheim-zziplib-6b6242e/zzip/plugin.c /path/to/gdraheim-zziplib-6b6242e/zzip/stat.c /path/to/gdraheim-zziplib-6b6242e/zzip/write.c /path/to/gdraheim-zziplib-6b6242e/zzip/zip.c --package=zziplib --version=0.13.72 --onlymainheader=zzip/mmapped.h --output=zzipmmapped
writing 'zzipfseeko.html'
writing 'zzipfseeko.html'
writing 'zzipfseeko.docbook'
writing 'zzipfseeko.docbook'
cd /tmp/build-libzzip/docs && /bin/mv zzipfseeko.docbook zzipfseeko.xml
cd /tmp/build-libzzip/docs && /bin/mv zzipfseeko.docbook zzipfseeko.xml
mv: zzipfseeko.docbook: No such file or directory
make[2]: *** [docs/zzipfseeko.xml] Error 1
make[2]: *** Deleting file `docs/zzipfseeko.xml'
make[2]: *** Waiting for unfinished jobs....
[ 97%] Linking C executable zzipwrap
cd /tmp/build-libzzip/zzipwrap && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/zzipwrap.dir/link.txt --verbose=ON
/Library/Developer/CommandLineTools/usr/bin/cc -O3 -DNDEBUG -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/zzipwrap.dir/zzipwrap.c.o -o zzipwrap  -Wl,-rpath,/tmp/build-libzzip/zzipwrap -Wl,-rpath,/tmp/build-libzzip/zzip libzzipwrap-0.13.0.72.dylib ../zzip/libzzip-0.13.0.72.dylib /opt/local/lib/libz.dylib /opt/local/lib/libz.dylib
writing 'zziplib.html'
not found in anchors: ['zlib', '_open', 'ZZIP_FILE', 'ZZIP_DIR', 'buf32k', 'inflateInit', 'EOF', '__zzip_find_disk_trailer']
writing 'zziplib.docbook'
writing 'zziplib.html'
not found in anchors: ['zlib', '_open', 'ZZIP_FILE', 'ZZIP_DIR', 'buf32k', 'inflateInit', 'EOF', '__zzip_find_disk_trailer']
writing 'zziplib.docbook'
cd /tmp/build-libzzip/docs && /bin/mv zziplib.docbook zziplib.xml
writing 'zzipmmapped.html'
not found in anchors: ['munmap', 'zzip_file_header_to_data']
writing 'zzipmmapped.docbook'
cd /tmp/build-libzzip/docs && /bin/mv zziplib.docbook zziplib.xml
mv: zziplib.docbook: No such file or directory
make[2]: *** [docs/zziplib.xml] Error 1
make[2]: *** Deleting file `docs/zziplib.xml'
make[2]: *** Waiting for unfinished jobs....
cd /tmp/build-libzzip/docs && /bin/mv zzipmmapped.docbook zzipmmapped.xml
writing 'zzipmmapped.html'
not found in anchors: ['munmap', 'zzip_file_header_to_data']
make[2]: Leaving directory `/private/tmp/build-libzzip'
make[1]: *** [docs/CMakeFiles/htm.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
writing 'zzipmmapped.docbook'
make[2]: Leaving directory `/private/tmp/build-libzzip'
cd /tmp/build-libzzip/docs && /bin/mv zzipmmapped.docbook zzipmmapped.xml
make[2]: Leaving directory `/private/tmp/build-libzzip'
make[1]: *** [docs/CMakeFiles/man.dir/all] Error 2
[ 97%] Built target zzipwrap
make[1]: Leaving directory `/private/tmp/build-libzzip'
make: *** [all] Error 2
make: Leaving directory `/private/tmp/build-libzzip'

I guess the whole problem is just somewhere in proper specification of dependencies of manually-defined targets.

[mojca]

Btw: It would be really nice if the html documentation could be automatically installed into $prefix/share/doc/zziplib/ (either by default or with an additional option). At the moment we copy all the files "manually" to the destination inside a package manager. (Ok, with a script of course, but it still needs to be set up manually once.)

[gdraheim]

I can not reliably reproduce the problem but I saw it a few times and I can see in the build log that the xml doc targets appear twice.

[ 71%] Built target zzxordir
[ 72%] Linking C executable zzip
[ 73%] Linking C executable zzdir
[ 75%] Generating zzipfseeko.xml
[ 76%] Building C object test/CMakeFiles/zzipself.dir/zzipself.c.o
[ 78%] Linking C executable zzipsetstub
[ 79%] Linking C executable unzzip-mix
[ 80%] Linking C executable zzobfuscated
[ 80%] Built target zzip
[ 82%] Generating zzipfseeko.xml
[ 82%] Built target zzdir
[ 83%] Building C object bins/CMakeFiles/unzzip.dir/unzzipcat-zip.c.o
[ 83%] Built target zzipsetstub
[ 84%] Generating zziplib.xml                   # <<<<<<<<<<<<<<<<<<<
[ 84%] Built target zzobfuscated
[ 86%] Linking C executable zzipself
[ 86%] Built target unzzip-mix
[ 87%] Generating zzipmmapped.xml               # <<<<<<<<<<<<<<<<<<<
[ 89%] Building C object bins/CMakeFiles/unzzip.dir/unzzipdir-zip.c.o
[ 90%] Building C object zzipwrap/CMakeFiles/zzipwrap.dir/zzipwrap.c.o
[ 90%] Built target zzipself
[ 91%] Generating zziplib.xml                  # <<<<<<<<<<<<<<<<<<<
[ 93%] Generating zzipmmapped.xml              # <<<<<<<<<<<<<<<<<<<
[ 95%] Built target zzipwrap
[ 95%] Built target unzzip
[ 97%] Generating zzipshowme
[ 97%] Built target sfx
[ 98%] Generating htmpages.tar
[100%] Generating manpages.tar

So there seems to be some dependency generation being wrong but I am not sure how to find that or if it would be better to look for a workaround since the problem seems to come up when the "mv x.docbook x.xml" line is run in parallel. That is the last step in the "Generating x.xml" target.

[gdraheim]

I have modified the install/rules so that the site/ generation is now locally in the build-dir and not delayed to install-site time. That should allow the intended result.

-- Installing: html site to /usr/local/share/doc/zziplib
-- Installing: html pages to /usr/local/share/doc/zziplib/man
-- Installing: man3 pages to /usr/local/share/man/man3