Improve documentation browsing

[DanilYachmenev]

Here is a big but important one, believe in you @zli00185 Ask questions in the issue if anything is unclear

Goal: We would like to make it easier to browse / search the documentation in docs For now, we just go https://github.com/sorrentum/sorrentum/tree/master/docs and browse, or grep with bash

We would like to publish the docs with something fancier, something like:

• Sphinx https://docs.readthedocs.io/en/stable/tutorial/
• https://www.mkdocs.org/

GitHub supports automatically publishing things with https://docs.github.com/en/enterprise-server@3.6/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site

To make searching easy we could use https://www.algolia.com/

The outcome should be a project proposal Gdoc with some ideas

• Only open-source, it should be easy to use
• I would start from browsing the suggested solutions, read how they work and describe in a Gdoc how to use them in our case, what steps would it require and what we will achieve by implementing them
  • I'd personally start from the 1st suggested since it seems already adjusted to Git
  • Do not wait to read all the sources altogether - publish results 1 by 1 so we can discuss in the process

FYI @gpsaggese @samarth9008 @jsmerix

[zli00185]

Solution 1 Abstract: Read the doc could create online documentation from Sphinx project in Github. Here is an example of what it looks like. (source link) Steps:

1. Build a Sphinx project in GitHub repository.
2. Importing the project to Read the Docs.
3. Build the HTML documentation.
4. Trigger a build from a pull request and see the preview of the documentation
5. Adding the configuration file, a file named .readthedocs.yaml in base folder that customize the building process. (This step is optional until September 2023 but recommend)
6. Version update

Pro:

1. RTD could generate browsing traffic analytics and browsing search analytics
2. RTD will automatically regenerate the documentation and make the updated version according to the update in repository.
3. RTD offers built-in search functionality

[zli00185]

Is there any step that I should further describe in proposal?

[gpsaggese]

• Any idea on how to make the documentation searcheable once published?
• Can you point to a few examples of how it will look like?

[zli00185]

Does the outcome proposal Gdoc have to be formal?

[DanilYachmenev]

Does the outcome proposal Gdoc have to be formal?

no need to be formal in words but be structured and straight to the point it should be something that we can work together on afterwards (like extending, discussing points etc)

[zli00185]

Solution 2 Abstract: Mkdoc is a static site generator where file are written in markdown and configured with one YAML configuration file. (source link) Step:

1. Install Mkdoc with pip install mkdocs and create project with mkdocs new my-project
2. This will create a configuration file named mkdocs.yml, and a folder named docs where place all documentation source files.
3. After adding all the document, configuration file could provides customization like theming and changing the Favicon Icon.
4. Build the site with mkdocs build
5. Depoly the doc to Github with mkdocs gh-deploy

Pro: Provides more extensive customization options than RTD

How to make Mkdoc searchable without third part tool: https://github.com/squidfunk/mkdocs-material/blob/master/docs/setup/setting-up-site-search.md# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/

Examples: https://mkdocs-macros-plugin.readthedocs.io/en/latest/pages/

[zli00185]

Do I need to create a PR for the proposal gdoc?

[DanilYachmenev]

Do I need to create a PR for the proposal gdoc?

No need yet IMO lets discuss everything here until there is anything to commit to the codebase

You can make a PR if you may need to do some code changes to demonstrate smth

[zli00185]

Okay. So far I feel like Mkdoc is a more straightforward and lucid solution to me since RTD requires to build a Sphinx project first. I am still searching around for the difference between RTD and Mkdoc but I don't think there will be any huge difference on the final product besides customization. For searchable functionality, RTD has built-in search function, and I also find some open-source tutorials about setting up site search for Mkdoc using third party plugins.

[DanilYachmenev]

@zli00185 well a Gdoc with your findings would've done a good help, just post a link here in the issue so we have a doc that is easy to work on

also extrapolate on GP questions for Sphinx as well so we could compare fully

[zli00185]

Gdoc

[DanilYachmenev]

Gdoc

nice!

For now I'm more for Mkdoc since it is

• more customisable
• more straightforward to implement

Although 1 question: 1 of the mentioned RTD pros is automatic documentation update is it also available by Mkdoc?

[zli00185]

Gdoc

nice!

For now I'm more for Mkdoc since it is

• more customisable
• more straightforward to implement

Although 1 question: 1 of the mentioned RTD pros is automatic documentation update is it also available by Mkdoc?

Just confirm it is also available in Mkdocs.

[DanilYachmenev]

@zli00185 tbh for me it is still hard to comprehend how the final form will look like without a draft example Let's start from a prototype for Mkdoc

Implement the described steps, get a link to the formatted documentation and show the final look File a PR from a new branch if needed

[zli00185]

I build a prototype with two md files from sorrentum docs folders on my Github repository and it should be available online now. I didn't add any configuration so it is all in default setting. So far the searching functionality is workable, but I am still figuring out why the image is not showing. @DanilYachmenev Should I further improve the layout and theme of the current prototype?

[DanilYachmenev]

@zli00185, nice one! I could access the website

couple moments:

• lets deal with screenshots display, it is important
• can we remove default index.md file and put a README there instead?
  • also pls check if we can control the order of displayed documentation on the generated website and if ref links are working
• Check if doc updates are being pulled to the website automatically. If not, describe the process on how to pull the changes in docs to the website
• lets do a couple tweaks with the layout, theme, etc. Just do a couple that you consider interesting just to display the functionality

After done, lets also build the same size prototype for RTD so we can compare

[zli00185]

lets deal with screenshots display, it is important

Image display issue is fixed. We need to change the image display code in md file. For example, <img src="Buildmeister_process_figs/image1.png" style="width:3.40625in;height:1.82292in" /> need to be ![Alt Text](Buildmeister_process_figs/image1.png) (Just suggest, this might be a good warm-up issue)

I updated the md file in the Github repository and online docs should be updated too.

[DanilYachmenev]

@zli00185

• I do not see screenshots in the prototype website though, could you add some docs with it there?
• What I do not like actually is how the code blocks look like - they are almost indistinguishable from the other text. Could you pls try to make it more visible, as like in the current docs?
  • 
• And yep, see my other suggestions above, do them as well pls

[zli00185]

can we remove default index.md file and put a README there instead?

• also pls check if we can control the order of displayed documentation on the generated website and if ref links are working

I am pretty sure index.md could be replaced or modified, and the order of each md files could be changed in configuration file

Check if doc updates are being pulled to the website automatically. If not, describe the process on how to pull the changes in docs to the website

Just confirm it with the image code update. It will automatically update the library after git push.

[DanilYachmenev]

Just confirm it with the image code update. It will automatically update the library after git push.

do it in your repo/PR and confirm that it is necessary we will do it if we decide to go with Mkdocs for now we need to get a good looking example that will demo that it displays docs correctly

[zli00185]

• I do not see screenshots in the prototype website though, could you add some docs with it there?

The Buildmeister process page contain screenshot at Notification system section. It should be visible now.

[DanilYachmenev]

@zli00185 how's the progress with RTD?

[zli00185]

@zli00185 how's the progress with RTD?

I could probably upload the RTD library in two days. But before that I am still trying to figure out changing the theme for Mkdocs. There is a more popular theme called Material but I currently have some issue activate it. If we don't have to worry about theme at this point I could move on to build RTD.

[DanilYachmenev]

@zli00185 do not spend too much time on some actions if they are not crucial and take more than ~1 hour if nothing is found them its OK, just mention it in the report message

the higher priority is to see both sources and then we can decide if we need additional research to choose one of them

[zli00185]

Here is a prototype for RTD without sorrentum docs. Current issues:

1. RTD doesn't support markdown file, and I encounter issue activate extension for markdown.
2. After convert md file to rst file, the server could not recognize the page after rebuilding.

I am still searching for workable solutions for either issue.

[DanilYachmenev]

Here is a prototype for RTD without sorrentum docs. Current issues:

1. RTD doesn't support markdown file, and I encounter issue activate extension for markdown.
2. After convert md file to rst file, the server could not recognize the page after rebuilding.

I am still searching for workable solutions for either issue.

So RTD requires everything to be converted in their own format? If so, lets try to find a way to convert md files to the desired format however, it should be easy and native

spend no more than 1 hour on dealing with RTD and switch back to Mkdocs customization If we can make code blocks more visible IMO no need to spend more time on RTD

[zli00185]

All previous issues solved. Here is the final prototype for RTD from my Github repository. One thing about RTD I find usefull is it allow us to create different version of the library base on PR.

[DanilYachmenev]

@zli00185 nice! I personally like RTD more it is also in-line with documentation for other projects that I've seen

Is it much more complicated to implement than Mkdocs? You mentioned some problems with RTD implementation so I want to understand how crucial they are

Could you pls update Gdoc with links to prototypes and update info about pros and cons

[zli00185]

Could you pls update Gdoc with links to prototypes and update info about pros and cons

Gdoc updated

Is it much more complicated to implement than Mkdocs?

It actually is not difficuilt at all. The tuteriol of extension for Mkdown file does not mention putting the extension into requirenment.txt file (which is a simple but crucial step) and that is why it kept failing on servicer. Once I figure that out, it is as straightforward as using Mkdocs.

[zli00185]

The theme for mkdocs has been changed to Material, which I personally believe is a better theme than mkdocs. Here are the prototype for RTD and Mkdoc.

[DanilYachmenev]

@zli00185, good job! pls take a look at questions I left at Gdoc so we make the last clarifications before making a proposal about the better option

[DanilYachmenev]

Now we need to choose 1 among the 2 options tested by @zli00185 and continue moving documentation there.

Both options have document browsing feature and quite nice visual interface. Below you can find prototypes and distinctive pros and cons (if 1 is present in 1 option in means that another does not have it):

1. Read the Docs (RTD)

Prototype link

Pros:

• Can be edited through GH PRs
• In my experience, a lot of other projects have documentation in RTD so its kind of conventional
• Can generate browsing traffic analytics and browsing search analytics (not so important IMO but nice to have)

Cons

• Has non removable ads at the left bottom of the library

2. Mkdocs

Prototype link

Pros:

• Document browsing is nicer (search preview is displayed)
• Provides more extensive design customisation and IMO looks cooler in current implementation

You can see full report and discussion in Gdoc

So in short, RTD would be much easier to update, while Mkdocs will be much nicer to use. My pick is Mkdocs - all for the final user even if it means a bit more pain for us.

WDYT @gpsaggese?

[DanilYachmenev]

@gpsaggese could you ptal at the message above so we can proceed?

[gpsaggese]

Can we move the gdoc https://docs.google.com/document/d/1lOS9s88WdXnjbv5pm0C7ZXqacvfnAWu0Yum_jRLF_lQ/edit to a dir corresponding to the bug in https://drive.google.com/drive/u/1/folders/1LXwKpmaFWJI-887IoA50sVC8-dw_1L8I? You might need to copy / paste since once the file is created in a different account is difficult

[gpsaggese]

Adding @jsmerix since this is part of the push towards better documentation.

Nicely done @zli00185

Questions:

• What are the advantages wrt to going to https://github.com/cryptokaizen/cmamp/tree/master/docs?
  • IMO the search, the fact that looks nicer and more professional, what else?
• Can we publish this under GitHub?
• Personally RTD looks more familiar since most of the open-source projects use that
• We can pay for the service https://about.readthedocs.com/pricing/

[DanilYachmenev]

Can we move the gdoc https://docs.google.com/document/d/1lOS9s88WdXnjbv5pm0C7ZXqacvfnAWu0Yum_jRLF_lQ/edit to a dir corresponding to the bug in https://drive.google.com/drive/u/1/folders/1LXwKpmaFWJI-887IoA50sVC8-dw_1L8I? You might need to copy / paste since once the file is created in a different account is difficult

https://drive.google.com/drive/folders/1Ch2XyqWKAdj8wc-MolGxK-cWYwgwN9y8?usp=drive_link

[gpsaggese]

Thx. Also can we convert to Google doc instead of sh*tty Word and remove the Micro$oft junk.

[DanilYachmenev]

• What are the advantages wrt to going to https://github.com/cryptokaizen/cmamp/tree/master/docs?

The ones you've mentioned are the most important ones.

Apart from them both tools:

• have additional plugins like embedding dynamic charts and diagrams, gathering usage analytics etc
• allow automatically update links and references
• allow to have multiple versions of documentation (might be useful in future where we might have differing instructions for different software)

• Can we publish this under GitHub?

Yes, and the changes can be pulled automatically afaik