Feature: modernize API documentation

[f18m]

This is a WIP PR. See mailing list for more details. See current rendering at: https://f18m.github.io/libzmq/ Current known issues: links in MANPAGES are weird (show some ".html" text) Additionally this MR requires squashing prior merging.

• migrate from the old, unmaintained "asciidoc" tool to the new "asciidoctor" generator; note that Asciidoctor currently works on libzmq docs usin the "compat mode" (see https://docs.asciidoctor.org/asciidoctor/latest/migrate/asciidoc-py/); replacement of section titles with the new Asciidoc syntax is required to get rid of the compat mode.
• remove the need of "xmlto" utility to create the manpage output; use asciidoctor for that
• add HTML output support to the doc/Makefile by using asciidoctor
• change API documentation files extension from .txt to .adoc to make it more explicit that they are Asciidoc-encoded (as a bonus several IDE plugins will autodetect the .adoc format as Asciidoc)
• remove asciidoc.conf: asciidoctor does not support that; this also required replacing the macro linkzmq into all documentation pages
• add a new Github action CI do deploy the static HTMLs produced by Asciidoctors

[f18m]

hi @bluca , @sappo , I marked this PR as ready: I did my best to check every possible HTML page and most of the manpages. Everything looks good to me.

Additional steps compared to the PR original description:

• converted the asciidoc-py legacy syntax to the "modern" Asciidoc syntax supported by Asciidoctor and other tools
• removed references to the "xmlto" and "a2x" tools from the build systems: Asciidoctor can convert the documentation directly to e.g. pdf (via extended converters) and anyway there was no code/target for using "xmlto" and "a2x" tools anyway

Additional tests I've done:

• I tested the cmake build on Linux, with/without Asciidoctor installed
• I tested the automake build on Linux, with/without Asciidoctor installed
• I tested manpage installation

After merging I suggest to go in the github "Settings" and then "Pages" and choose as "source" the "Github Actions" option. See screenshot taken from my fork:

[bluca]

How can this integrate into wikidot where the rest of the documentation is (previous versions, other projects, etc)

[sappo]

@bluca it won't. Older docs will still be available in wikidot. This PR will provide the latest docs on gh pages and eventually readthedocs where newer version will then be published to.

[bluca]

That's problematic, search engines will always point to the old URLs as they have been around for long, so it would need redirecting, but that can't happen if the previous versions are lost

[f18m]

That's problematic, search engines will always point to the old URLs as they have been around for long, so it would need redirecting, but that can't happen if the previous versions are lost

A couple of considerations:

• the API documentation published on wikidot can still be updated if there is interest in doing so... I hope it's just a matter of enabling with few more Makefile lines the generation of .xml files out of Asciidoctor. Then there is a Perl script at https://github.com/zeromq/ztools/blob/master/zmqapi/xml2wd.pl in charge of doing the XML->wikidot syntax My question is: is there still interest? Currently http://api.zeromq.org/ mentions v4.3.2 released back in 2019...

• If you agree I think readthedocs.io would be a much better replacement to host libzmq API documentation for multiple versions. After this PR I can volunteer to work on integrating libzmq in readthedocs.io and push there the docs for past versions of libzmq as well. This would become a more modern/usable replacement of the whole http://api.zeromq.org/ website I believe (there was some email about that also on zeromq-dev mailing list)

• I think that if we switch to readthedocs.io most search engines will pick up the change pretty quickly; this might improve current situation: right now the search results for most libzmq API point to very old versions of libzmq... e.g. if I search for "zmq_connect" my Google shows as top result the http://api.zeromq.org/3-2:zmq-connect page, from libzmq 3.2... I'm not sure why it happens... but moving to a newer platform will make latest versions more prominent I guess (just because of SEO of modern websites)

[bluca]

I have no opinion on where things are hosted and in which format, as long as we don't break things - if google searches suddenly start showing the wrong website or dead pages, then that's a problem. If it can be made to work with some redirects or aliases or whatnot, then it's all fine by me

[f18m]

I have no opinion on where things are hosted and in which format, as long as we don't break things - if google searches suddenly start showing the wrong website or dead pages, then that's a problem. If it can be made to work with some redirects or aliases or whatnot, then it's all fine by me

@sappo do you think it would be possible to setup some redirection from api.zeromq.org to a zeromq.readthedocs.io website (which of course does not exist yet)? Github pages enabled on libzmq (as suggested in this PR) are a nice way for developers to quickly check their documentation updates but for the zeromq userbase I think it's good to have docs by released version, just like readthedocs.io does, so I keep suggesting to go in that direction.

Looking at https://www.wikidot.com/doc-modules:redirect-module, it seems to be possible to setup such kind of redirect... but I don't have the credentials for the zeromq wikidot website of course...

[bluca]

I can do that change too, I have admin access to wikidot as well

[f18m]

I can do that change too, I have admin access to wikidot as well

ok wonderful. Then do you think anything else is needed in this PR before merging ? After this one I can contribute the readthedocs.io integration if that's the agreed solution going forward...

[bluca]

The dependencies in packaging/ need to be updated to the new doc build tools before this can be merged, otherwise we'll get no manpages in the packages built on OBS

[f18m]

The dependencies in packaging/ need to be updated to the new doc build tools before this can be merged, otherwise we'll get no manpages in the packages built on OBS

done, but I'm not sure how to test it. I found this OBS page: https://build.opensuse.org/project/monitor/network:messaging:zeromq:ci:zeromq:libzmq:PR-4607

for this PR, but in the "Monitor" tab it shows "Unavailable Build Results." I could find results only for the "release-stable" package: https://build.opensuse.org/project/monitor/network:messaging:zeromq:release-stable but also there on the row "libzmq" all results are "Broken"...

[bluca]

some cache was broken, try to rebase and force push? hopefully it will actually build this time

[bluca]

also squash the commit, no need to have it separate

[f18m]

done, the branch should be aligned to latest master, all commits squashed... let's see

[bluca]

looks like asciidoctor is not available in a bunch of distributions: https://build.opensuse.org/package/show/network:messaging:zeromq:ci:zeromq:libzmq:PR-4607/libzmq

[f18m]

looks like asciidoctor is not available in a bunch of distributions: https://build.opensuse.org/package/show/network:messaging:zeromq:ci:zeromq:libzmq:PR-4607/libzmq

I think in some distros (Centos/SuSE) the package is often named "rubygem-asciidoctor"... but that comes typically from EPEL or similar repositories... do you know if e.g. EPEL is enabled or can be enabled during the OBS build?

[bluca]

Yes EPEL is enabled: https://build.opensuse.org/projects/network:messaging:zeromq:git-draft/meta I guess the spec file needs alternative dependencies depending on where it's being built?

[f18m]

Yes EPEL is enabled: https://build.opensuse.org/projects/network:messaging:zeromq:git-draft/meta I guess the spec file needs alternative dependencies depending on where it's being built?

I pushed an attempt to install "rubygem-asciidoctor" on RHEL/SuSE and just "asciidoctor" everywhere else. I don't have the possibility to test these many distros myself so I'll watch OBS results and iteratively try to fix all reds... at the end I can re-squash all commits together...

[f18m]

Yes EPEL is enabled: https://build.opensuse.org/projects/network:messaging:zeromq:git-draft/meta I guess the spec file needs alternative dependencies depending on where it's being built?

I pushed an attempt to install "rubygem-asciidoctor" on RHEL/SuSE and just "asciidoctor" everywhere else. I don't have the possibility to test these many distros myself so I'll watch OBS results and iteratively try to fix all reds... at the end I can re-squash all commits together...

@bluca do you know if it's possible to enable

https://build.opensuse.org/package/show/devel%3Alanguages%3Aruby%3Aextensions/rubygem-asciidoctor

during the zeromq OBS build?

[bluca]

Those are 'devel' projects, IE they are used for development of new packages/versions, and should not be used when doing release builds - so no, however from there it should get included in SUSE releases

[bluca]

I think you have to use rubygem(asciidoctor) on suse rather than rubygem-asciidoctor? https://build.opensuse.org/projects/openSUSE:Factory/packages/rubygem-asciidoctor/repositories/standard/binaries/x86_64/ruby3.2-rubygem-asciidoctor-2.0.20-1.1.x86_64.rpm

[f18m]

I think you have to use rubygem(asciidoctor) on suse rather than rubygem-asciidoctor? https://build.opensuse.org/projects/openSUSE:Factory/packages/rubygem-asciidoctor/repositories/standard/binaries/x86_64/ruby3.2-rubygem-asciidoctor-2.0.20-1.1.x86_64.rpm

thanks that worked well. Now only reds are Centos8, RHEL7 and RHEL8. Centos8 has some trouble installing ruby itself (message is "nothing provides ruby") while RHEL7 and RHEL8 seem to be lacking EPEL (message is "nothing provides rubygem(asciidoctor)").. any chance EPEL can be enabled in RHEL7 and RHEL8? I googled around about centos8 but so far no luck. Centos8 mentioned by OBS is Centos8 Stream right?

[bluca]

yes it's stream - is epel compatible with rhel too? I am not really familiar with the rhel side of the world

[f18m]

yes it's stream - is epel compatible with rhel too? I am not really familiar with the rhel side of the world

yes, EPEL is fully compatible with RHEL (and Centos Stream as well btw)... for me it's the opposite: I'm mostly familiar with RHEL side of the world, less with Debian side :)

[bluca]

actually it's already configured, both centos and rhel are using the epel repository too

[f18m]

actually it's already configured, both centos and rhel are using the epel repository too

then I'm a bit lost... using virtualbox I just run a Centos8 stream, and did successfully:

dnf install -y epel-release  # to get EPEL 
dnf repolist  # this one shows "EPEL8 and EPEL Modular 8" being installed
dnf install -y rubygem-asciidoctor
asciidoctor --version # shows version 2.0.20

not sure if OBS gives some possibility to debug on the Centos8 system? or print e.g. the output of "dnf repolist" somewhere?

[bluca]

no that's not possible - maybe rubygem(asciidoctor) is suse specific, and for rhel it needs rubygem-asciidoctor ?

[f18m]

no that's not possible - maybe rubygem(asciidoctor) is suse specific, and for rhel it needs rubygem-asciidoctor ?

using both rubygem(asciidoctor) and rubygem-asciidoctor syntax, the error basically does not change. Do you think it's OK to skip manpage build for Centos8 and RHEL7/RHEL8 OBS builds? I feel most of RHEL users will take the zeromq RPM coming from EPEL: https://packages.fedoraproject.org/pkgs/zeromq/zeromq/ And my understanding is that the SPEC file used by EPEL is always customized and is not the one in upstream repo... so they have the freedom to enable/disable doc building in EPEL regardless of what we do with OBS.

[bluca]

Which package provides ruby(rubygems) on a real installation? We can try a substitution

[f18m]

Which package provides ruby(rubygems) on a real installation? We can try a substitution

ruby(rubygems) is provided by the RPM named just rubygems. This is on my vbox Centos8 stream:

Last rpm query with --provides shows that "rubygems" RPM is providing "ruby(rubygems)". I will try the same on RHEL8 but I'm pretty sure it's identical. Note that "rubygems" RPM comes from the AppStream repo, not even from EPEL...

[bluca]

I managed to get it working for centos with a hint from an obs admin, the ruby stuff is in a "module" so it has to be enabled with ExpandFlags: module:ruby-3.0, rhel still doesn't like it tho, will keep trying for a while before giving up

[f18m]

I managed to get it working for centos with a hint from an obs admin, the ruby stuff is in a "module" so it has to be enabled with ExpandFlags: module:ruby-3.0, rhel still doesn't like it tho, will keep trying for a while before giving up

oh I see -- nice catch. This mechanism of "modules" is still something I'm not used to. Also AFAICT it should have been introduced from RHEL 8 upward. So it's strange to see RHEL7 failing (with EPEL enabled). But thanks to your efforts I see that now both Centos8 and RHEL8 are green in OBS, very nice.

[bluca]

Yeah so you can remove the different name of the dependency, as it seems it's not necessary, and also you can exclude it from rhel/centos 7 or lower so that the build doesn't break, then this can be merged

[f18m]

Yeah so you can remove the different name of the dependency, as it seems it's not necessary, and also you can exclude it from rhel/centos 7 or lower so that the build doesn't break, then this can be merged

done, even if the RHEL7 build in OBS keeps failing, but this time with the same error failure that I see also in libzmq "master" branch: nothing provides python3 needed by

[bluca]

yeah you can ignore that, at some point I'll look into it

[sappo]

@f18m @bluca great job on this PR. When this new docs have proven themselves we can also introduce them to czmq.

[f18m]

@f18m @bluca great job on this PR. When this new docs have proven themselves we can also introduce them to czmq.

Thanks, about czmq: I will save somewhere the script I used to mass convert the asciidoc from legacy to modern format, so we can use that for czmq as well

I'm now looking at readthedocs integration...