How about moving the manual to Gitbook?

[ousado]

see: https://www.gitbook.com/ features: https://www.gitbook.com/blog/features examples: https://www.gitbook.com/explore It has all the benefits - many export formats, would allow for much simpler contributions - and even offers JS plugin support.

[nadako]

I played with it a bit and I must say I strongly support this idea. TeX looks too scientific and hard to deal with to an average guy (like me), and I personally would contribute much more if the source format was Markdown (see my C# target manual yet to be converted to tex).

GitBook editor is quite straightforward, supports Git-flavored markdown and has instant preview which IMO is very important (look at this screenshot http://take.ms/jO7vF). And the files can be managed by git as we do now, with branches, PRs and all this.

It also has EPUB, MOBI and PDF exports with custom stylesheets, and the thing itself is extensible with plugins (see http://rustbyexample.com/ for an example of compiler integration in the doc, which we could also implement using @clemos's try.haxe.org or @waneck's new development).

@Simn @Gama11 @markknol @andyli what do you think about this? I'd volunteer to help migrating to this if you all agree (that means hacking our markdown exporter to generate a structure that would be easier to import into gitbook, e.g. generate the SUMMARY.md file as described in Gitbook's manual: http://help.gitbook.com/format/chapters.html).

[andyli]

I'm all for avoiding the use of TeX. I used TeX for writing my thesis and a few papers. I understand its power in handling complex formula and citation. But interestingly, even scientific journals (e.g. IGI Global) and big conferences (e.g. SIGGRAPH) are moving away from TeX that they either deprecates their TeX templates or simply only accept .docx submissions... Anyway, TeX is not designed for publishing web page, which is currently the most important target for us.

Also, I would like to point out that the formatting of our current EPUB, MOBI output isn't perfect (particularly when it come to tables). So if gitbook can export better versions of those, great!

[nadako]

Okay, I guess I'll just take the initiative and try to prepare fully usable gitbook project so we can scout for any troubles.

[fullofcaffeine]

That reminds me of this issue I created in mid 2014 - https://github.com/HaxeFoundation/HaxeManual/issues/38 :)

I'm all for using a better format for writing the manual, too. Good to know we're moving to GitBook! Hopefully it will foster collaboration.

Cheers.

On Fri, Sep 18, 2015 at 1:46 PM, Dan Korostelev notifications@github.com wrote:

Okay, I guess I'll just take the initiative and try to prepare fully usable gitbook project so we can scout for any troubles.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/HaxeManual/issues/194#issuecomment-141533356 .

[markknol]

I think markdown is great and I hope it will encourage more contributions. If we still can have enough flexibility in formatting/creating links/references etc that would be nice.

About structure, I think it would be nice to have the main sections in a separate file, I dislike the idea of having each page in separate document (as the current export is right now) for editing. Is that possible?

[markknol]

Maybe stupid question, but can you still host the content on our own website? Most examples I clicked were on gitbook.

[indolering]

I strongly support moving to Markdown, but I don't see GitBook as compelling unless you are planning on publishing a book. There are a lot of Markdown-based static site generators, including some with GUIs that connect to Github directly. A generic site generator would allow you to use custom plugins and add arbitrary functionality that you won't get from GitBook.

[ousado]

@indolering As far as I'm aware the TeX based solution has been chosen for that reason, being able to publish books. Regarding plugins, GitBook allows for that, unless you're thinking of a feature set not provided.

[nadako]

So I hacked md converter to generate SUMMARY.md and it already looks quite cool: https://www.gitbook.com/read/book/nadako/haxe-manual

It builds automatically on pushes using a github webook, similar to travis/appveyor (https://github.com/nadako/HaxeManual)

[nadako]

It also built PDF, MOBI and EPUB files (using default templates/settings) which you can check out here: https://www.gitbook.com/book/nadako/haxe-manual/details (press the "Download" button).

[nadako]

Heh, it even has "accept donations" flag ("Donations allow readers to encourage you to write books by making a donation between $3 and $100.") which could be a nice way for people to support documenting haxe with money.

[nadako]

We can use this plugin https://plugins.gitbook.com/plugin/mermaid-2 for to render all kind of diagrams (see https://github.com/knsv/mermaid)

[nadako]

@markknol sure, gitbook is basically just a tool (https://github.com/GitbookIO/gitbook), its website is like "Heroku for books", so we can entirely move everything to our platform.

[nadako]

So I propose we configure our custom build using GitBook and custom templates, and hook it up on travis to auto-update the manual on haxe.org.

Also, they have desktop editor app with instapreview (https://www.gitbook.com/editor) which seems very nice.

[Simn]

I'm not opposed to this, but I would still like to have some notion of compile-time check for section references and general formatting. Does the new approach allow for that?

[nadako]

I think references are checked by gitbook builder (I'll check it out when I get to it again), can you elaborate about those general formatting checks?

[Simn]

TeX has a syntax so when you mess up e.g. a listing definition you'll probably get errors. With markdown on the other hand you don't really notice the problem until you look at the rendered content. I suppose this is somewhat mitigated by the fact that markdown syntax is much easier overall, so it's harder to mess it up.

Either way, since everyone seems to agree this is the right path I won't stand in your way. I don't expect a huge increase of contributions from this, but I don't see any harm either.

[nadako]

It is also mitigated by the instant preview feature of most markdown editors, so you edit md on the left and see the result on the right, it's really hard to mess up.

Okay then, I'll prepare a branch with travis build and make a PR. I'm not really familiar with ebooks though, but I'll research that topic.

[indolering]

TeX has a syntax so when you mess up e.g. a listing definition you'll probably get errors. With markdown on the other hand you don't really notice the problem until you look at the rendered content. I suppose this is somewhat mitigated by the fact that markdown syntax is much easier overall, so it's harder to mess it up.

Someone still has to accept the pull requests, I would think that would catch most errors. You could also setup a bot to spider the links and catch any breakage.

[MatthijsKamstra]

Change this HaxeManual to markdown is a great idea. It's easy to write and gitbook creates a nice website (desktop version is good, on ipad it starts with a collapsed menu and the phone version is terrible). BUT....

I have been documenting my progress (Haxe / Javascript / Node.js) with Gitbook/Markdown and I thought it would make it easy to contribute.

And I can't make it any easier as it is:

• source on github
• markdown
• website on github
• edit this page (links to the md source)
• disqus

But there is little (perhaps I expected too much) contribution or maybe it's the subject or what-ever-reason-you-can-think-off, but the real reason is that people just don't contribute that much.

https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)

I had to add analytics to see if it was used by anyone.

[indolering]

@MatthijsKamstra I strongly support adding (moderated) comments to any FOSS documentation, especially for programming languages. Half of the value of the PHP documentation comes from the comments. There are plenty of people who just don't have time to rewrite the primary documentation.

[MatthijsKamstra]

I was just trying to make it as-easy-as-possible to contribute and if I can think of another way I will add that too. I forgot about the PHP documentation, but you are so true.. the comments make it better than the original documentation. What I miss in a lot of documentation are working code examples. I remember when I started to learn AS2 I just copied the working example (script kiddy) and modified it till it worked for me.

Although this HaxeManual is a little different, it could benefit from comments.

[indolering]

I was just trying to make it as-easy-as-possible to contribute and if I can think of another way I will add that too.

Yeah, I worked on this problem before and the Github based workflow for the documentation is close to ideal. Pure wiki's make it really easy to contribute but it doesn't work on a small scale and for technical documentation you typically need some level of review. You try to fill the gap by creating an anonymous contribution system that submits automated pull requests.

I forgot about the PHP documentation, but you are so true.. the comments make it better than the original documentation.

Maybe you should open a new ticket to add a comments section to the manual and API pages?

What I miss in a lot of documentation are working code examples.

Agreed! I really dislike the API documentation for this reason and I would love to see links to try.haxe.org for all examples. That would also make it easy for reviewers to ensure that the example actually works. But this sounds like it needs a new ticket....

[nadako]

So, I am looking into this again and I see there's actually a lot of work was done to ease managing the current TeX version and we need to get it ported to GItbook. Gitbook is actually awesome in the sense that it's quite easily extensible using blocks and filters, so we need to write a gitbook plugin that provides the same functionality as we currently have. Let's make a list of what's need to be ported:

1. TODO markers with comments: collect those and filter out from the output
2. Missing content checker: create a list of sections without content, unless they have special tag
3. Reviewed marker: collect a list of unreviewed sections
4. Modified marker: ???
5. Including code: we can either keep storing .hx files separately and use include-codeblock plugin to include them, or we can do vise-versa: move code blocks inside markdown files and extract them for CI testing
6. Flowchart generation: it's probably a good idea to move these away from tex tools to something simplier as well, but I'll have to research on what to actually use for that, as the only related tools I know are DOT graphs and Graphviz to visualize them.

Other things to research:

• embedding gitbook content in haxe website (as far as I understood, the "official" gitbook way is to override css files and inject html content to predefined places, but that may not be enough for us, so we may have to dig into custom templates which is not very well documented/supported).
• generating ebooks: I personally don't know much about ebooks, but apparently gitbook supports them well enough. it seems to have necessary metadata configuration and separate css styling for all formats

Another nice thing we could benefit from is the edit-link plugin that provies a nice "edit this page" link that points to a github fork&edit interface.

Anyway, I was playing with gitbook-the-software without the gitbook-the-service and managed to set up an example "book" that simply builds with Travis CI and it's quite easy, you can look at the example repo: https://github.com/nadako/playground-gitbook-travis (and the auto-pushed from travis book output: http://nadako.github.io/playground-gitbook-travis/).

cc @Simn (did I forget anything?)

[nadako]

@indolering Sorry, I totally missed your https://github.com/HaxeFoundation/HaxeManual/issues/194#issuecomment-142385145. I was thinking about "simply" rolling out a custom markdown-based generator, but I came to conclusion that we'll just end up in a stripped-down and hacky version of gitbook and that looks like a waste of development resources, also we still want to export to ebook formats. Gitbook looks customizable/extensible enough to me right now (as I described in a comment above). But surely it's an open discussion!

[Simn]

• The current markdown converter collects all definitions and generates a dictionary from them.
• You can lose the reviewed/modified markers, it's not very important.
• I would really prefer to keep the .hx files separately. We currently test them on the CI thanks to @Gama11's work and we should definitely keep that.

[nadako]

The current markdown converter collects all definitions and generates a dictionary from them.

Gitbook uses a different approach - we write the dictionary (glossary) ourselves and then it highlights terms automatically (see https://help.gitbook.com/format/glossary.html). I currently generate the glossary file similar to old dictionary with a convert tool.

I would really prefer to keep the .hx files separately. We currently test them on the CI thanks to @Gama11's work and we should definitely keep that.

Yeah that's why I mentioned an option to extract fenced code blocks from markdown with a plugin and test them. I'm unsure myself what is the better option though.

[Simn]

I have a feeling this is one of these issues where everyone happily discusses changes that ultimately never see the light of day. I'm not blaming anyone and I certainly know @nadako in particular is trying, but I don't really like this state of uncertainty.

So let's talk reality: Could we get this migration done before the end of the month?

[nadako]

I am not sure, since I have yet to think of something for flowcharts (probably using the hxdotgraph thingy). But anyway in the end that I'm hoping for, there will be a script to convert a current version of tex manual to a gitbook, so it's not blocking issue.

[Simn]

Dot graphs are very ugly compared to what we have currently. I originally made the flowcharts using it, but fortunately @SmilyOrg showed up and fixed that.

Let's just keep the tikz files and use the generated images for the gitbook version. The flowcharts never change anyway, so it's a matter of generating them once.

[nadako]

Are you talking about the generated images or the input format?

[nadako]

The ugliness, I mean. :)

[SmilyOrg]

If possible, it would also be nicer to use vector images (svg/pdf/somethingelse) for PDFs and printed text at least, so you don't get ugly low dpi printed raster graphics. Should be like a command line switch for pdflatex or whatever is being used to compile them :)

[Simn]

@nadako: The generated images of course.

[nadako]

Okay, I worked on it again today and managed to set-up book auto build+deploy with Travis. I also wrote my own plugin for including code assets from separate files (because include-codeblock is really weird) and it's quite simple once you get used to "guess the API" :)

Everything is in this branch right now: https://github.com/nadako/HaxeManual/tree/gitbook (see gitbook and gitbook-tutorial folders) and it auto-deploys via Travis to gh-pages: http://nadako.github.io/HaxeManual/. The MD files are generated from current TeX using a modified (or, rather, quickly hacked) convert utility.

I am not sure whether the current markdown layout is good enough for editing and further maintainance. For example, I think we should generate trivia, since and other commands as gitbook blocks (e.g. {% trivia "Bleh" %}Some text{% endtrivia %}) and render them with our custom plugin, instead of hand-writing markdown headings and quotings.

I'll look into that, but anyway I would like current content contributors to look at the MD files and comment about their structure, naming schema and general maintainability. cc @Simn @markknol @andyli @Gama11

[Simn]

My main problem is that I can't "see" the order and hierarchy or the markdown files. The only reason this works at all is because we have a consistent naming scheme.

I'll have to look into how gitbook editing works before I can comment on the maintainability. I'm assuming this is somehow dealt with via https://github.com/nadako/HaxeManual/blob/gitbook/gitbook/SUMMARY.md.

[Simn]

Also I'm too stupid to set this up on gitbook.

[nadako]

Yeah, SUMMARY.md file defines the structure of the book, it's basically it's ToC.

[nadako]

We could name those files with chapter number prefix if you like

[Simn]

It's okay if the editor sorts it properly.

I tried setting up gitbook but couldn't figure out how to deal with the fact that in your repository the md files are in a sub-directory. It worked out when I moved everything to top-level.

Also the current SUMMARY.md loses the hierarchy, there are just two levels.

[MatthijsKamstra]

I love it, great work @nadako !

I only miss the "edit this page" (https://www.npmjs.com/package/gitbook-plugin-edit-link) and discus (https://github.com/GitbookIO/plugin-disqus)

• edit for easy contributions (will fix also this: https://github.com/HaxeFoundation/HaxeManual/issues/194#issuecomment-171206926)
• discus, so users can ask questions about the documentation (or add examples: https://github.com/HaxeFoundation/HaxeManual/issues/194#issuecomment-147795864)

Perhaps google analytics would be good also: so we know which part is visited most

[markknol]

Can we keep the current menu structure, in the current menu we have more depth in the navigation, now everything seems flattened. For example

• 'Target Details > Javascript > Debugging JavaScript'
• 'Language features > pattern matching > introduction'

[nadako]

Yeah, I'll look into this.

However, after messing with Gitbook for some time, I found out that its editor doesn't seem to use plugins for its insta-preview functionality, so our fancy {% trivia %}, {% codeasset %} etc tags won't render correctly in a preview, which is a bit frustrating. I'm now even thinking about making a "simple" custom markdown build system.

Also, I think we could re-organize .md files in some folder hierarchy, because having that amount of files in one folder looks scary.

[indolering]

Any updates on this? I've been holding off on edits....

[nadako]

Sorry, I didn't have time for this lately. Feel free to contribute to the TeX files for the time being, we'll convert the latest version to MD when it's ready.

[Simn]

I'll go ahead and close this now as I don't see it happening. It's not that I'm opposed to it, but someone would just have to do it.