automated build of protobuf documentation missing

ArcEye commented 6 years ago

Issue by mhaberler Thu Jun 16 06:41:49 2016 Originally opened as https://github.com/machinekit/machinekit/issues/970

this is an anchor issue for integrating the machinekit protobuf docs into the website

it's a sub-project of https://github.com/machinekit/machinekit/issues/924 which is getting a tad long, so let's focus related work here

my overall idea is:

the protobuf definitions are all important and need to be documented - ideally the same way like all the rest, 'Edit this page' link inlcuded on a per-proto-file basis
there is a neat tool - protoc-gen-docs - which aids in generating docs from proto files - see input example and resulting output
protoc-gen-docs is already part of the jekyll website build container and running on jenkins.machinekit.io

So to integrate the protobuf docs build into the website I guess we need to:

add the protobuf repo into the machinekit-docs repo as a subtree (just as with mk/mk - no magic, explained here)
add a Makefile step which runs protoc-gen-docs on the proto files, generating asciidoc; that is in place already and just needs to be integrated, and proper place for the output found
adapt the Mustache template to output proper asciidoc - this file needs translating from markdown to asciidoc syntax ; the current output formats do not include asciidoc yet
think through how 'Edit this proto file' links can be massaged into the output
link the protobuf docs page in a warm place (likely near manual pages)

The build process then would be like so:

update the protobuf git subtree
run the protobuf-docs make target which does the protoc-gen-docs job
run the website build

I will take care of the jenkins work to achieve that, but a sharing work with the template work would be great

ArcEye commented 6 years ago

Comment by ArcEye Thu Jun 16 08:57:52 2016

I have updated my pull to include protobuf.

Will see what running make generates and go from there.

First thought is that the whole Table of Contents section at the top of the template is unnecessary. So long as the file contains sections and subsections, just having :toc: beneath the first section header will produce it automatically

ArcEye commented 6 years ago

Comment by ArcEye Thu Jun 16 10:47:15 2016

Pretty much done

Assuming the :ignore-front-matter: is still being added by the docs build, asciidoc.mustache now renders a asciidoc like so https://github.com/ArcEye/machinekit/blob/master/src/machinetalk/proto/build/doc/machinetalk-protobuf.asciidoc (if you view it as 'Raw' the TOC will be rendered - if you have the asciidoc plugin in your browser)

Which converts into a html with TOC like so https://github.com/ArcEye/machinekit/blob/master/src/machinetalk/proto/build/doc/machinetalk-protobuf.html (you will need view it here http://mgware.co.uk/temp/machinetalk-protobuf.html why on earth github refuses to just display html docs properly, I don't know)

ArcEye commented 6 years ago

Comment by ArcEye Thu Jun 16 13:55:11 2016

Finally found a way to make all types into links, so that clicking the type within a table, takes you to either the entry in the table of standard data types or the special type definition header.

One way which worked perfectly in asciidoc, ceased working once the document was converted to html :stuck_out_tongue_closed_eyes:

Links above updated to view results

ArcEye commented 6 years ago

Comment by ArcEye Thu Jun 16 14:04:49 2016

think through how 'Edit this proto file' links can be massaged into the output

Edit this page is meaningless for this document. Since it is completely generated, you have to either edit the template or the actual source code.

Suggest a rider on the document to that effect, unless we can think of a way to disable the Edit this page link when this document is displayed.

ArcEye commented 6 years ago

Comment by mhaberler Thu Jun 16 15:03:31 2016

hey, superb progress!

"Edit this page" might refer to the template - does that make sense?

otherwise it'd be great to have an edit link per-proto-file, that would make it super-easy to touchup on them/document them - the proto edits would go to mk/machinekit-protobuf

ArcEye commented 6 years ago

Comment by ArcEye Thu Jun 16 15:34:52 2016

"Edit this page" might refer to the template - does that make sense?

Not really, since the template has no control over the content, just the format and order it is displayed in. It is available for PR like any other file

otherwise it'd be great to have an edit link per-proto-file, that would make it super-easy to touchup on them/document them - the proto edits would go to mk/machinekit-protobuf

If you can think of a way to do that's great. {{page.edit-repo}} and {{page.path}} are not going to point there.

The situation is just the same for the generated man pages, editing the page is pointless as the component is in another repo and needs to be subject of a proper PR, not just a document presentation tweak.

I think we just have to be realistic, if someone actually knows something about protobuf or components, they should be able to make a PR in the normal way.

ArcEye commented 6 years ago

Comment by ArcEye Thu Jun 16 15:40:10 2016

The main thing that needs doing in the .proto files is adding descriptions in a way that will get picked up by proto-gen-doc.

That needs doing by someone who actually knows about them.

You will note that every Description field in the tables is currently empty.

Use /** */ or /// comments to document your files. 
Comments for files go at the very top of the the file. 
Comments for enumerations, messages and services go before the message, enumeration or service definition. 
Comments for fields, enum values, extensions and service methods can go either before or after the definition. 
If a documentation comment begins with @exclude, the corresponding item will be excluded from the generated documentation.

ArcEye commented 6 years ago

Comment by mhaberler Fri Jun 17 06:49:30 2016

congratulatios - https://github.com/estan/protoc-gen-doc/pull/171 !

ArcEye commented 6 years ago

Comment by mhaberler Fri Jun 17 06:54:11 2016

I defintely will take on cleaning up the protos and adding proper comments

being a lazy bastard I just thought we can work in an additional per-proto link to like for instance https://github.com/machinekit/machinetalk-protobuf/blob/master/src/machinetalk/protobuf/rtapicommand.proto

anyway let's get this into the build process first and rearrange deckchairs second - what do I need to do?

ArcEye commented 6 years ago

Comment by ArcEye Fri Jun 17 07:39:45 2016

congratulatios - estan/protoc-gen-doc#171 !

It is a good system and surprisingly easy to get professional results Us moustachios have to stick together :smile:

ArcEye commented 6 years ago

Comment by ArcEye Fri Jun 17 07:50:44 2016

anyway let's get this into the build process first and rearrange deckchairs second -

I will do a PR for the template shortly, but it will be subject to change to get the descriptions in

what do I need to do?

I would suggest just take one file for now and annotate as per the quoted text, 3 posts above.

Try a general description in a /** */ block at the top of the file then comments behind /// before all functions, enumerations, messages, services, fields, enum values, extensions and service methods.

That will give a known format to work with and we can see what proto-gen-doc makes of it and how best to set the template to display it.

ArcEye commented 6 years ago

Comment by ArcEye Fri Jun 17 08:22:17 2016

Doing some experiments.

message Emc_Traj_Linear_Move {

    option (nanopb_msgopt).msgid = 103;
    /// description of something
    required MotionType   type = 10;
    required EmcPose      end = 20;
    required double        vel = 30;
    required double ini_maxvel = 40;
    required double        acc = 50;
    required bool    feed_mode = 60;
    required int32 indexrotary = 70;
}

will produce description of something in the description field for MotionType

So contrary to the docs, the description must appear immediately above (before) the field it is describing.

Putting the same comment below indexrotary produces nothing

ArcEye commented 6 years ago

Comment by ArcEye Fri Jun 17 08:56:15 2016

More experimentation

File descriptions MUST start at line 1, col 1

They can either be block commented with /** */ or a single /// if just one line If you want a new line in the text, must double space or they will run into each other

I will PR the amended template

Following this format produces:

selection_013

ArcEye commented 6 years ago

Comment by mhaberler Fri Jun 17 10:04:20 2016

super!

thinking how to do this in jenkins

as a first step I added machinetalk-protobuf as a subtree to the docs dir: https://github.com/mhaberler/machinekit-docs/tree/protobuf-subtree

commands how it was created + how to update are in the commit message

@ArcEye - if this is to your liking, I'll make it a PR

the Makefile would do the git subtree pull ... then create the protobuf asciidoc file as a temp file, compare it to the one in the tree, and update it if they dont (to save unnecessary rebuilds)

sounds about right?

ArcEye commented 6 years ago

Comment by mhaberler Fri Jun 17 14:55:43 2016

I must have botched that branch above

this looks more like it: https://github.com/machinekit/machinekit-docs/pull/166

frontmatter still missing

ArcEye commented 6 years ago

Comment by mhaberler Sat Jun 25 14:49:54 2016

it seems to me we're next to done with the plumbing part - the editing of the protobuf messages itself of course not withstanding?

ArcEye commented 6 years ago

Comment by ArcEye Sat Jun 25 15:01:28 2016

Looks like it, they seem to build fine

ArcEye commented 6 years ago

Comment by ArcEye Wed Jul 13 08:24:01 2016

@strahlex Don't know if you saw this, as you were away at the time

Whilst you are working upon the protobuf files, be aware of the documenting conventions as above at https://github.com/machinekit/machinekit/issues/970#issuecomment-226710134

If you use them, anything thus marked will be automatically rendered in the docs

machinekit / machinekit-cnc

automated build of protobuf documentation missing #41