Building documentation packages from the new machinekit-docs

[luminize]

There's still work to get static HTML or PDF documentation. Generating packages that go together with the Machinekit packages. IMO no rush, but let's not forget this.

I think this means building against a PR to the machinekit-docs

[zultron]

Does the machinekit-docs build system still work to build docs? (I'm too lazy to actually try, sorry. :P )

[luminize]

There's still a Makefile, so I'll have to try manually.

On 13 Jan 2015, at 16:49, John Morris notifications@github.com wrote:

Does the machinekit-docs build system still work to build docs? (I'm too lazy to actually try, sorry. :P )

— Reply to this email directly or view it on GitHub.

[gmoccapy]

I tried with the makefile, but was not able to get it working, No rule to run, no autogen no config, so i was not able to preview the changes I made locally.

Norbert

[luminize]

Hi Norbert, This is broken. So that's to be expected. Most important is that we can view in Github. Bas

On 10 Dec 2015, at 11:19, Norbert Schechner notifications@github.com wrote:

I tried with the makefile, but was not able to get it working, No rule to run, no autogen no config, so i was not able to preview the changes I made locally.

Norbert

— Reply to this email directly or view it on GitHub.

[bobvanderlinden]

@luminize Thanks for creating the issue. I gave this a try by converting all asciidoc files to markdown and generate the readthedocs page based on @luminize readthedocs experiment. The result is viewable on http://machinekit-docs.readthedocs.org/en/latest/. It is also viewable and editable on Github: http://machinekit-docs.readthedocs.org/en/latest/

readthedocs uses the directory structure to show the links on the sidebar. At the moment it is a bit cluttered. I'd think it's possible to create a custom sidebar, but it might also be a good idea to structure the files in a way that fits the sidebar (easy to locate which file is where).

The following command was used to convert all .asciidoc files to .md:

for file in $(find . -name '*.asciidoc'); do asciidoc -b html -o - $file | pandoc -f html -t markdown_github | sed 's|\.asciidoc|.md|g' > ${file%.asciidoc}.md; echo ${file%.asciidoc}.md; done

[luminize]

@bobvanderlinden , I'm speechless You are right on all points. We would better work towards getting the structure straight. Going to bed happy !

[mhaberler]

@bobvanderlinden - that is an amazing result for a one-liner - I think that is already better usable than the raw asciidoc github repo!

with a bit of preparation to increase the conversion rate (links, tables, images) that looks like a viable route to me. Do I get this right - both website and docs would use the same markdown source format eventually?

btw - how do we proceed with the non-english versions? Frankly except for the french version, they are mostly unusable. What would the feelings be towards dropping the non-english versions?

[bobvanderlinden]

Thanks! I'm not quite sure where to start on the structure though. I never realized how huge the documentation actually was.

@mhaberler Yes, everything is created from the .md files. The sidebar is at the moment generated from the structure of the files and the webpages are generated using the contents of the markdown files and mkdocs.

I agree on the Spanish and French docs. Readthedocs does seem to have some support for languages, so those could still be introduced, but I have a feeling those will get out of sync quickly. Dropping them will create less confusion I think.

The one-liner still left some garbage in the files as HTML. Like footnotes still have HTML around them.

There were a few files that failed to build, but I couldn't really figure out what was wrong. The python generator crashed with an exception (no line or anything). Those files in particular are really big, so I dropped them for the moment and get a feel for what the rest of the docs looks like.

[the-snowwhite]

Wow Great work ..! This makes the documentation come alive and very appealing to work with.

I would be very nice if some consideration also was put into making it 5 - 10" tablet / pad friendly. (I'm not able to test that as I don't have one available atm ).

But it would be great for bedside / armchair reading / /editing / authoring ....

@mhaberler --> https://github.com/machinekit/machinekit/issues/840#issuecomment-165848703

About kernel (uboot) and other building processes they can be documented in text style like here: https://eewiki.net/display/linuxonarm/DE0-Nano-SoC+Kit#DE0-Nano-SoCKit-Bootloader:U-Boot

Where you then can copy paste the commands into you console.

or in simple single function scripts where the working kernel , u-boot versions url's etc can be grouped, tested and maintained via variable strings. (somewhat ala like this):

https://github.com/the-snowwhite/mksoc/blob/master/scripts/Divided_scripts/build_kernel.sh https://github.com/the-snowwhite/mksoc/blob/master/scripts/Divided_scripts/build_uboot.sh

[mhaberler]

well frankly this looks viable to me - what are your opinions on proceeding this route?

let's think further steps:

• conversion process: best would be to reanimate the Makefile in the docs repo, and then tack on the conversion process (this should help with cross-file links, and be the step where to split out per-language docs, that separation is already in the build); I can look into that but not realistically before wednesday
• with that build we can tune the conversion by iterating builds before we commit to drop asciidoc and go for polishing the markdown version, like fixing the asciidoc pages which do not convert right now
• static manual pages - HAL and RTAPI man pages: not sure how to go about these, but I guess these need to be converted once as well
• autogenerated manual pages: not sure where the format stands there, but that needs to tacked onto the machinekit build followed by an upload of markdown to the common git repo
• conversion cleanup post commit to markdown: I guess we need to split up work - like reviewing/fixing up per section - it'd be unfair to dump that onto a single person
• fixing the navigation/toplevel structure: not sure but I guess post commit to markdown

is this where we are heading?

[ArcEye]

For some strange reason, a lot of the links within documents have changed to linkname.md and are now dead.

This presumably comes from the sed 's|\.asciidoc|.md|g' section of the script, but why most have been resolved in the conversion, but some have not, is peculiar.

See the documentation matrix for most examples, but they exist elsewhere too

[luminize]

@mhaberler wrote:

is this where we are heading?

Yes i think so. I would be willing to drop asciidoc right away. Let's do some really small scale proof of concept experiments, no going off into the sky as per here https://github.com/machinekit/machinekit-docs/issues/48 so we can start working in parallel. Is the format really a part in the equasion of getting info out of machinekitinto machinekit-docs?

I have already prepared stuff w.r.t. getting all into machinekit-docs from the machinekit.io Like building instructions for jessie etc. https://github.com/luminize/machinekit-docs/tree/jessie-instructions (need to polish commits). I prefer to be done once and for all with working in 2 places.

@the-snowwhite I think we can document similar when we put the instructions into code blocks, like here this has aways worked for me. Or do you mean something else?

[luminize]

@ArcEye I think the documentation matrix has to go. As are the table headers/footers I made for walking thru documents. So My take would be that we change the structure in bits, and do some manual cleanup. Doesn't mean it's a one person job. We can work in parallel with that.

[luminize]

I just did a very small test on the pointer of @bobvanderlinden on gitbook. Gitbook needs a SUMMARY.md file which specifies sections. A bit more work to maintain but an alternative we can keep alongside readthedocs.

See my SUMMARY.md in the root directory which produces this: https://luminize.gitbooks.io/machinekit-docs-mkdocs-test/content/getting-help.html

[luminize]

@mhaberler wrote

• conversion process: best would be to reanimate the Makefile in the docs repo, and then tack on the conversion process (this should help with cross-file links, and be the step where to split out per-language docs, that separation is already in the build); I can look into that but not realistically before wednesday

When I first ran sphinx-quickstart from within the docs, there was a Makefile being generated. This is the one so we might have to look a bit more into sphinx.

[gmoccapy]

@bobvanderlinden,

you are all too fast for me ;-) but I try to maintain the step with you. I just tried:

http://machinekit-docs.readthedocs.org/en/latest/.

I clicked user interfaces / gmoccapy and as expected got the documentation of the corresponding GUI. First I see, that some formating has been destroyed, as:

• Basic Configuration : The list of provided INI's is not any more a list
• all internal links seem to be dead
• some more small issues

So I decided, lets correct the "mistakes" and see how it works. So I clicked on the tob right link "Edit on Github", on that page I went to srv / gui and looked for a gmoccapy file. It is not there!!! Where are the original file now?

@All: On the previous discussion a read that nowbody does print the docu. So in that case, I am one of the nowbodys ;-) as I got no internet or telefon on my shop and that is the place where I do my rebuilding of machines. So IMHO it should be possible to print the documentation, or at least get it as pdf.

I waited long time to place the gmoccapy documentation in the machinekit repo, not because I am to busy, or too lasy. I just needed to get myself focused on how to do that. I missed a very simple tutorial like " If you want to add documentation do it this way, if you want to edit a page it is done that way" I am pretty sure, that many people do not touch the docu, because it has been not as user friendly, as it should be. So we will not only have to look at a format, but also at a WYSIWYG Editor to do so.

Norbert

[luminize]

Hi @gmoccapy,

Thanks for checking. Bob did a batch conversion and we should really check these points you mention. In short the conversion was from asciidoc to HTML to markdown. Since a direct way of generating markdown from asciidoc was not possible.

Ill have a look at the "edit on Github" you mention. What i can trace back is that this links back to bobs repository. Maybe that's the cause.

As far as the printing of documents goes. If I have my facts right, PDF as well as ePub (have to try that out) can be generated from various services. Plus you can very easily set this up yourself. I'll gather my notes and will show how I reproduce the readthedocs locally (serving in your own browser from a local server)

I'll add your points here too https://github.com/machinekit/machinekit-docs/issues/48

Bas

On 21 Dec 2015, at 17:58, Norbert Schechner notifications@github.com wrote:

@bobvanderlinden,

you are all too fast for me ;-) but I try to maintain the step with you. I just tried:

http://machinekit-docs.readthedocs.org/en/latest/.

I clicked user interfaces / gmoccapy and as expected got the documentation of the corresponding GUI. First I see, that some formating has been destroyed, as:

Basic Configuration : The list of provided INI's is not any more a list all internal links seem to be dead some more small issues So I decided, lets correct the "mistakes" and see how it works. So I clicked on the tob right link "Edit on Github", on that page I went to srv / gui and looked for a gmoccapy file. It is not there!!! Where are the original file now?

@All: On the previous discussion a read that nowbody does print the docu. So in that case, I am one of the nowbodys ;-) as I got no internet or telefon on my shop and that is the place where I do my rebuilding of machines. So IMHO it should be possible to print the documentation, or at least get it as pdf.

I waited long time to place the gmoccapy documentation in the machinekit repo, not because I am to busy, or too lasy. I just needed to get myself focused on how to do that. I missed a very simple tutorial like " If you want to add documentation do it this way, if you want to edit a page it is done that way" I am pretty sure, that many people do not touch the docu, because it has been not as user friendly, as it should be. So we will not only have to look at a format, but also at a WYSIWYG Editor to do so.

Norbert

— Reply to this email directly or view it on GitHub.

[bobvanderlinden]

@gmoccapy The resulting markdown code isn't what it should be yet. Like said before, it still keeps some HTML, which results in incorrect documents in some cases. The links are directly rewritten, it seemed to work fine for most links, but I can imagine there are errors.

If you find any more errors, could you link me the exact page that is causing the problem? I'd like to find all edge-cases and change the conversion script to cope with those cases.

As for the "Edit on Github" link, it is linking to the bobvanderlinden/machinekit-docs repository which shows the master branch as default. You need to look for the markdown branch, as that's the one that's generated for readthedocs. Additionally, I'd like to see "Edit on Github" linking directly to the file on Github, instead of just to the root of the Github repository. This is being worked on: https://github.com/mkdocs/mkdocs/pull/752

@luminize As for PDF and ePub. I'm now using mkdocs and not Sphinx to generate the documentation. mkdocs has good support for Markdown, whereas Sphinx has less support. mkdocs does not support generating PDF or ePub: http://read-the-docs.readthedocs.org/en/latest/builds.html#mkdocs

[gmoccapy]

@Bobvanderlinen,

thanks for your effort, I will go on working on that next year, as I finnaly got a new laptop ;--) and I have to get rit of the windows 8 and install all my development system on it.

But I will read about the development in background ;-)

Norber

Am 21.12.2015 um 23:30 schrieb Bob van der Linden:

@gmoccapy https://github.com/gmoccapy The resulting markdown code isn't what it should be yet. Like said before, it still keeps some HTML, which results in incorrect documents in some cases. The links are directly rewritten, it seemed to work fine for most links, but I can imagine there are errors.

If you find any more errors, could you link me the exact page that is causing the problem? I'd like to find all edge-cases and change the conversion script to cope with those cases.

As for the "Edit on Github" link, it is linking to the |bobvanderlinden/machinekit-docs| repository which shows the |master| branch as default. You need to look for the |markdown| branch, as that's the one that's generated for readthedocs. Additionally, I'd like to see "Edit on Github" linking directly to the file on Github, instead of just to the root of the Github repository. This is being worked on: mkdocs/mkdocs#752 https://github.com/mkdocs/mkdocs/pull/752

@luminize https://github.com/luminize As for PDF and ePub. I'm now using mkdocs and not Sphinx to generate the documentation. mkdocs has good support for Markdown, whereas Sphinx has less support. mkdocs does not support generating PDF or ePub: http://read-the-docs.readthedocs.org/en/latest/builds.html#mkdocs

— Reply to this email directly or view it on GitHub https://github.com/machinekit/machinekit-docs/issues/7#issuecomment-166441253.

[luminize]

@gmoccapy I have a test here: http://machinekit-docs-mkdocs-test.readthedocs.org/en/docs-revamp-1/ This is generated by sphinx, and because of that we have the possibility of getting for example epub (reading on your tablet) and pdf. See the pfd that got generated on readthedocs from this branch on a test repo note: no proper index (yet)

@mhaberler @bobvanderlinden I replicated bobs delete on unbuildable files with this commit https://github.com/luminize/machinekit-docs-mkdocs-test/commit/368431f334242a851b3284928862265a0638d718 and this commit https://github.com/luminize/machinekit-docs-mkdocs-test/commit/1a433bea5f6db69632b33316d4a07371297ca452

A look into the src directory revealed legacy makefiles and submakefiles. IMO these are all obsolete and can be removed? This could be the reason for the commits above. the first commit is trivial, obsolete indexes anyway, but the second one is about remapping. Any takers on the second one?

@mhaberler @ArcEye any reason why we shouldn't work on the machinkit/machinekit-docs repo?

[mhaberler]

@luminize - the way to fix this is to repair the underlying asciidoc errors in those documents (esp if they are key parts of the documentation) - I'll take that on for structure.asciidoc (which does not even display properly now either)

[mhaberler]

I looked at the build process as it was before we spun out machinekit-docs

there is zero point in trying a conversion without getting the build again. All the cross-linking of documents, link-checking, generating toc, cross-reference etc comes from there. A haphazard file-by-file conversion just to get some output right now will lead to a major loss of information which could be salavaged if we convert once a proper build works again.

It will be some work but cant get to it yet until later this week. If somebody else looks into defibrillating the build before I get around to it - fine. Do not proceed without it.

[luminize]

@mhaberler that would be wonderfull. The src/Master_* documents are for building separate documents if I'm correct. I see no need to keep that in tact. Do you?

regarding building. I'm a bit at a loss now... What do we want to build? I thought we aim to get docs viewable with less clicks etcetera. As a bonus we can have PDF built on readthedocs, or manually locally.

So what more needs to be built?

[mhaberler]

@luminize to understand what the build does, you'd need to run it and see the results

your best bet actually might be to clone the git.linuxcnc.org repo, and configure with --enable-simulator --enable-build-documentation, and run make; we then need to patch up machinekit-docs/Makefile to run again (which might require some backporting of src/Makefile fragments to build prerequisite intermediate files)

it will give you an idea what goes on in the docs build - this is not just about running asciidoc on a source file and be impressed with some html output ;)

[mhaberler]

once we have that build run going again, we can:

• fix up all asciidoc source files so they actually pass
• backport any valuable additions to the docs the linuxcnc folks might have made in the meantime (this must happen at the asciidoc level, once we commit to markdown it's too late)
• patch in @bobvanderlinden's asciidoc->markdown conversion changes into the build so a markdown tree is generated by the Makefile
• iterate on that state until we get the best conversion result achievable

only then we should cut the lines and switch to patch up/work on the markdown versions

[luminize]

clear. point taken. Since this issue is about the building itself, i'll post results on the other experimentshttps://github.com/machinekit/machinekit-docs/issues/48 where we can decide on what to do with cutting the lines after the documentation builds again.

[ArcEye]

@mhaberler @ArcEye any reason why we shouldn't work on the machinkit/machinekit-docs repo?

I am just going to sit it out until the waters clear, anything done until then is liable to be wasted, in the wrong format or whatever.

There are 2 issues,

• The readability / accessibility / auto updating of the documentation
• Its content

I am unclear exactly what is proposed for issue 1). Reversing the removal from sources and build from there? Put it into a separate builder with github hooks to auto build and update when amended?

Whatever method we will still have s**t, but in a gilded frame at the end of it unless 2) is taken care of.

I don't want to take ownership of the docs any more than Bas does, it is a thankless task, where everyone else gripes but does nothing themselves. If LCNC did not have JT, their docs would have gone south a long time ago.

If by co-operative effort we can knock them into shape, suitable documentation can be made a pre-requisite of PRs thereafter where appropriate and hopefully prevent the same situation a few years down the line.

[luminize]

@ArcEye wrote:

Whatever method we will still have s**t, but in a gilded frame at the end of it unless 2) is taken care of.

I agree fully. I’t about the content. And that needs a good overhaul. Period.

I have finally understood the building that @mhaberler mentioned. That needs to be done for making the source as best as we can get before making a (possible) conversion / cut. Like getting references in the asciidoc correct and checked by building it. So when time permits I’ll have a try understanding that process.

however, the above or current situation should not prevent any developer from starting to write about pending stuff like the CI and recent build stuff (but there’s lots more that is not documented). If there’s a script later on for converting, then that effort is not wasted. All that’s really needed is a new directory in the root of the docs, and some new files which describe the process. Everything can me put into better places as a whole in a later stadium with trivial effort if this is needed.

I don't want to take ownership of the docs any more than Bas does, it is a thankless task, where everyone else gripes but does nothing themselves.

Speaking for myself: I have no problem writing docs. But I’m not doing it alone anymore. I was very happy that you helped out the last year. But 3 people making commits to the docs is a little pathetic if you ask me. I have heard frequently last year “developers cannot write good documentation” thats BS in my world. It’s not about prose, Captals, points, comma's, headers or style of writing. It’s about getting the basics into docs so anybody understands. And then others can brush this up with style. If LCNC did not have JT, their docs would have gone south a long time ago.

If by co-operative effort we can knock them into shape, suitable documentation can be made a pre-requisite of PRs thereafter where appropriate and hopefully prevent the same situation a few years down the line.

That's the thing. The cooperative way. And there’s nothing standing in the way as it is now, a nice gilded frame does not prevent cooperation. I’ll stop preaching now :) .

Bas

[luminize]

for clarification of above https://github.com/machinekit/machinekit-docs/issues/7#issuecomment-166612994, so there's no misunderstanding: i'm not throwing the towel into the ring just yet.

[bobvanderlinden]

@mhaberler Yes, converting everything using the makefile would be much better indeed.

[mhaberler]

ok, I have a first cut of a wildly hacked standalone HTML build of the LinuxCNC documentation (NB: not the machinekit-docs repo)

https://github.com/mhaberler/machinekit-docs/tree/lcnc-docs-standalone-htmlformat

sudo apt-get install python-lxml
cd docs/src
make htmldocs

this should result in a working html tree (all links resolved etc) with the current style of the LinuxCNC docs in docs/html

now for going forward, my recommendation would be:

• take this as a base tree (I think it was generally better maintained than the mk docs)
• apply all relevant changes in machinekit-docs ontop of this
• assure the html build still works

Once there, we can tack on markdown conversion

sounds like a plan?

[mhaberler]

I added in a first cut on html->markdown conversion

the proper directory structure for jekyll is still missing and image links are likely broken

[luminize]

@mhaberler sound like the start of a plan. Are you proposing staring all over again and cherry-picking commits? Not a problem per se, if that's needed, it's needed, but I hate doing things twice, so let's make sure we won't have to do it a 3rd time next year :) first there must be a conversion from .txt to .asciidoc.

We need to have a better understanding of the structure we need (depends if we want to use jekyll/mkdocs/sphinx/readthedocs/gitbook/some/all or whatnot for eye pleasing reading).

I made https://github.com/machinekit/machinekit-docs/issues/48 is there stuff still missing we must experiment with?

[ArcEye]

first there must be a conversion from .txt to .asciidoc

That bit is quite simple. With double spacing etc, .txt converts quite well straight to .asciidoc.

See the man2asciidoc script in the /Maintainence dir

It just cats the man file output into a text file, but because of the formatting I used in it, it can be converted straight to asciidoc, using pandoc and the switches in the script

@luminize - Merry Christmas by the way ( you must be hiding from the kids and in-laws presently )

[luminize]

On 25 Dec 2015, at 15:57, ArcEye notifications@github.com wrote:

@luminize - Merry Christmas by the way ( you must be hiding from the kids and in-laws presently )

@ArcEye just had some spare minutes before going to dinner :) Merry Christmas to you and all others too.

[mhaberler]

"conversion" - I guess you mean rename .txt to .asciidoc and modify the directory structure of the linuxcnc-based branch to match the machinekit-docs structure

as for bringing in the changes which happened to machinekit-docs since the fork - that is a standard git rebase+fix the fallout job, no idea what "doing it twice" is supposed to mean

that is all fine, nor difficult but is irrelevant with respect to this work, as this focuses on a maximum conversion result - but then the few changes and new files could be brought in right now as well

"structure" - I guess this means the index and major chapters. Great if there is work done on that, but again disjoint from the automatic conversion job.

@ArcEye - yes we should explore the best possible conversion from groff to asciidoc and then whatever we choose; where is that Maintenance directory? lost track of what comp does.. generate troff? asciidoc?

[ArcEye]

where is that Maintenance directory?

https://github.com/machinekit/machinekit-docs/blob/master/machinekit-documentation/maintainence/man2asciidoc.sh

comp and instcomp generate groff man pages

I changed instcomp to add formatting which largely inserts extra spacing.

The net result is that when outputing a man page as text, you can convert that straight into .asciidoc with the stanza pandoc -s -S $compname.txt -t asciidoc -o $compname.asciidoc and you get an asciidoc which requires no further editing.

eg. https://github.com/machinekit/machinekit-docs/blob/master/machinekit-documentation/components/abs.asciidoc

[mhaberler]

hm, that deletes a lot of markup (fonts etc); playing around - this gives slightly better results but still unsatisfactory:

groff -mandoc -Thtml pid.9 | pandoc -f html -t markdown_github >/tmp/pid.md

still needs front matter like

---
layout: docs
title: The pid HAL component
prev_section: website
permalink: /docs/pid/
---

playing with other tools like doclifter.

[luminize]

closing this because of https://github.com/machinekit/machinekit/issues/924