Beautify docs

[BoPeng]

The github wiki is great for drafting our documentation but at this point we should produce better-looking pages with figures, pop-up examples etc.

[BoPeng]

We could also use gists to provide examples once we move our documentation to bopeng.github.io/sos.

[gaow]

Great move! Should we consider HUGO? https://gohugo.io/overview/introduction/

[BoPeng]

HUGO supports markdown, which makes things a lot easier, but I am not sure if its format is as limited as github markdown. Also, my biggest concern is syntax highlighting, namely if it uses codemirror or pygments so that we can syntax highlight our examples.

Gist can be a good idea for examples because we can edit them externally, but it does not support syntax highlighting of sos code.

EDIT: I see hogo supports pygments but not sure if it accepts our pygment extension for sos.

[BoPeng]

Another requirement would be easy to use navigation, because the github wiki side bar is not very helpful.

[gaow]

HOGO side bar looks great. I think it should also support mixture of MD and HTML so if we compile SoS script into HTML and plug into the MD (maybe streamline it via some scripts) we should be good.

Another option is jeklly which interacts nicely with github pages.

[BoPeng]

The homepage also need to be a bit more modern looking (e.g. http://immatics.com/), with icon, definition in large font, less text and more picture and youtube video.... Perhaps I should ask for some professional help.

[gaow]

Professional help 👍 My hope is that we do not have to give up on MD for the source part.

[BoPeng]

Jun tried to create the doc using jekyll, https://jma7.github.io/sos_wiki/ . Missing pieces include

1. sos pygments syntax highlighting
2. figure links
3. order of toc
4. Find a good looking theme (http://arkadianriver.github.io/spectral/ , http://themes.gohugo.io/theme/hugo-universal-theme/ ? The latter is for HUGO.)
5. More markdown feature/equivalence to warning, tip, etc
6. a home page.

For home page, I would like to have a HUGO style homepage with

LARGE LOGO/ BANNER

-----------
several icons, links to doc, download, example etc
------------

three side by side videos for 
[file format] [interactive] [workflow]

--------------
file format summary, and example

---------------
interactive use summary and example

--------------
workflow use summary and example

---------------
other information.

[jma7]

I tried to add the pygments into jekyll, which seems working fine on my desktop, please take a look at http://10.105.38.239/SOS/ This is running on my desktop. The keywords in SOS are highlighted.

However when I push the gh-pages branch to SOS github, I get the message with this link: https://help.github.com/articles/page-build-failed-config-file-error/#fixing-highlighting-errors So it seems pygments is not the supported highlighter on github.io. The supported highlighter is rouge, so maybe add SOS lexer to rouge? Right now, the markdown file at http://bopeng.github.io/SOS/ are using python style.

Just keep a record here, steps to add custom pygments file into jekyll.

1. git install pygments.rb
2. add the new lexer into gems/pygments.rb-0.6.3/vendor/pygments-main/pygments/lexers
3. In the same directory run python _mapping.py
4. in the same directory run: ./pygmentize -S default -f html > pygmentsSoS.css
5. add the link to css file in the header file. Change highlighter to pygments in _config.yml 6 use liquid style {% highlight sos %} then {% endhighlight %} in markdown file.

[BoPeng]

So we cannot use syntax highlighting unless we learn some ruby? The 'spawn a process slowness' criticism to pygments does not stand because github hacked pygments to pygments.rb to make it work on github....

I think the best way to proceed would be giving up markdown with scripts and use ipynb for all documentation with code. All we need is a script to convert ipynb to html and publish the html files to github.io.

[BoPeng]

Another possibility is to find a md to html converter and convert our markdown files to HTML using pygments before publishing to github.io.

[BoPeng]

I have converted all markdown docs to ipynb and moved them to doc/. I suppose we only need a bash file to batch convert them to github.io.

[gaow]

So are we now directly editing docs in Jupyter and save as ipynb, and we'll convert ipynb to HTML to post online? For example if I am to make changes, should I work directly on the ipynb? This format is less transparent than markdown. I just worry trivial changes such as cell number (not the contents) will trigger in ipynb file update; or do we still have some markdown version and we convert to ipynb from markdown via some command? Also with HTML generated this way, can we still have sidebar like the HUGO website?

[BoPeng]

There is no perfect solution as we have found that github.io does not support pygments so there is no easy way to syntax-highlight SoS code in markdown. Also, because of the rapid evolvement of SoS, it is very difficult to keep the documentation up to date in MD format so being able to execute documentation is very important. In the end we decided to use ipynb so that users can run the examples in the documentation directly from the public sos server. The static pages would be converted from ipynb using jupyter nbconvert.

The jupyter markdown is actually easier to use than github markdown, at least it supports HTML which makes customization a bit easier. The jupyter notebook initially only support forward-style workflow, I have changed it to be highly compatible to the command line version to better demonstrate features of SoS.

To answer your questions:

1. Jun Ma is working on a script to "publish" ipynb file .
2. Yes, I have made the mistakes to use random numbers in ipynb. We should correct that, but even then the ipynb files would change due to, for example, version of R used to produce.
3. There would be no side bar and the website is still being worked on. I believe that we can easily create sidebars from HTML files if we really need them though.

One trick that I am using is to run "run all" from Jupyter to make sure all cells run correctly. This checks correctness of code and make sure all cells are sequentially numbered.

[BoPeng]

BTW, there is a toc extension for Jupyter. https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/toc2 , which could help export a TOC during nbconvert.

It is actually pretty cool:

1. pip install jupyter_contrib_nbextensions
2. conda install -c conda-forge jupyter_nbextensions_configurator
3. jupyter contrib nbextension install --user
4. jupyter nbextension enable toc2

Or go to server_url/nbextensions, select Table of content, and the notebook would have a "Navigation" menu and you can open a menu panel by clicking an icon.

[BoPeng]

Thanks to Jun, the ipynb to html process now works like this.

1. The website is on the master under directory website.
2. The documentation is located at doc, with subdirectories examples, documentation and tutorial
3. development/release publish converts documents under doc to HTML format under the website directory, pushes the change to master, and pushes the subtree doc to gh-pages.

Remaining issues:

1. We need to find a tool to check dead links, I used blc before with success but I cannot use it to check our homepage because of the way it is written.
2. How do we update homepage? (For documentation index, I see js/sos_script.js)
3. Where to put figures? website/img?
4. How to cross link from .ipynb?
5. TOC and home button.

[BoPeng]

How to adjust the font of the homepage? I like the font and layout of http://beakernotebook.com/index, perhaps we can copy some of the ideas?

[gaow]

It is great to have the ipynb to html process streamlined! I'm thinking in our documentation we should make this release script a good example for people to organize their research -- they analyze data in Jupyter, and release it to gh-pages for book-keeping and sharing with collaborators.

I volunteer to make this page of doc, as I'm going to learn / adopt this myself anyways. Thanks @jma7 !

[gaow]

Also, should we keep the website HTML and figures only in gh-pages not in master? This will keep our main repo cleaner.

[BoPeng]

Good idea. We have update_toc.sos under example now, which is obsolete but can still be used as an example. We can put release over there later.

I just added toc to the pages, which is actually nice since the toc is float. It would be really nice for us to add a ".." (up, toc) button or a drop down list of all documentations so that we can easily navigate across pages.

[BoPeng]

Yes if you can figure out some easy way to do that.

The problem is that we need to keep doc in master because they will be included in the notebook docker image. It would be difficult to publish the docs if we have to switch and sync between master and gh-pages. The current setup allows us to do everything in master and publish the website with a single command (./release publish (perhaps should be renamed to publish_doc or doc, and on a related topic perhaps we should allow - in workflow name)).

The repo is not clustered that much because website is a subtree that is pushed to gh-pages. You can think of having master and gh-pages both under your hand and feels better about it. :-)

[gaow]

@BoPeng do you think a project whiteboard would help organzing issues? It can add cards, and convert github issues to cards. Issues with tags are good, but organizing via whiteboard may be helpful, too.

https://github.com/BoPeng/SOS/projects/1

Thanks for the clarification to the 2 branches. I'll have to experience with them and see if I can possibly make an improvement :)

[BoPeng]

Depending on if we can make full use of it. Now that we have three people working on the same project, perhaps it is a good idea to let each other know who is working on what...

Let us try it for a few days.

[BoPeng]

Can a note be claimed by someone as I am doing it?

[gaow]

It looks we can edit all notes no matter who created it. You can either assign or we can volunteer. For notes I create I just assigned myself via @

[BoPeng]

Hmm, I like the "convert" to issue function because it allows us to discuss tasks there.

[gaow]

Yes, and you can also do it backwards -- create issue and drag it to the board.

[gaow]

Someone just pointed to me that we do not have to use gh-pages branch to publish to github.io, as long as we are willing to put the entire website under master/docs. See under settings:

So we'll have to rename website folder to docs and use other names for the doc. This way we can stick to a single branch as we keep html files there anyways. Not sure if you want to adopt this convention.

[BoPeng]

Great! This would make things a lot easier for us. Could you make the change on website and development/release?

I do not mind having raw ipynb file with the html file in the same directory so you can simply merge the doc and website directories.

[gaow]

Sure, let me do this now -- but the setting requires that you do it on your end. Also you'd want to delete the gh-pages branch on github. I'll let you know when docs are renamed

[gaow]

Ok I've updated the repo structure and the release script. Now:

1. Notebook source for HTML pages are under docs/src
2. Published HTML pages are under docs/doc
3. The entire website is now moved from website to docs
4. gh-pages branch is no longer touched

@BoPeng please make the change from your settings to point the website to master/docs, and delete from the github repo the gh-pages branch, and see if it works. I did not test the release because I still cannot get toc2 to work properly ... But I think my changes to the release script is correct.

[BoPeng]

Done. I will test and update release if necessary.

[jma7]

@BoPeng This is what I mentioned to you on Tuesday, I read it somewhere.....

[BoPeng]

Things changed fast. I am almost certain this master/docs business was recent.

[gaow]

I tried this master/docs out in one of my project. I got an email "Page build failure" because apparently github tried to use jeklly to compile my project. It seems we should create a file called .nojekyll to avoid it. Just did it.

[BoPeng]

Just got the same email, and was wondering what to do. Thanks.

[gaow]

@jma7 would you kindly share your hack along with instructions on how to keep TOC on the left side of pages? I'm not able to ensure that so any release from my end would be problematic ...

[jma7]

It should work now by doing ./release update-website, would you please try again.