Need a good documentation portal for testng

[ayushchatur]

TestNG Version [Latest]

Note: only the latest version is supported

Hi , So far the test ng documentation looks like that of 80s. A new documentation portal is much required which shall have following benefits

• Better Structuring
• Fathomable view
• Latest content

and much more

cc: @krmahadevan

[krmahadevan]

@freakerhere - Thanks for following up on this. Are you willing to open up a PR for this ? We can start off with a very small change to get a feel of how the new UI would look like and perhaps take it from there ?

@email2vimalraj - I remember we spoke on similar lines as well. Care to chime in?

@juherr @cbeust - What is your thoughts on giving a new face lift to our documentation portal ?

[cbeust]

Always happy to brainstorm about better documentation but I'd like the discussion to be more focused.

For example, specifically, what kind of information is missing? What are the suggestions to make that data easier to find?

[email2vimalraj]

Hi Krishnan,

Thanks for bringing me to this conversation. My idea is to use the markdown files to create contents which will automatically produce HTML files and can be served through our hosting provider. I’m thinking of using GatsbyJS with our own simple UI design.

Also can leaverage GitHub actions to build and deploy automatically whenever the content is updated in GitHub.

Please let me know what you think.

Thanks Vimalraj

[krmahadevan]

@email2vimalraj - In terms of look and feel and the overall approach, do you think you could perhaps show a small demo implementation ? That could perhaps add more visibility into your proposal. Also it would be great if you could also list out what are the prerequisites in terms of software / libraries etc., that need to be installed in order to support this.

@cbeust What are your thoughts ?

[juherr]

Fyi, the documentation is already served by Github Pages. So you can directly write markdown and the file will be translated.

[email2vimalraj]

Yes will share a prototype this weekend.

Julien, so that means, testng.org is mapped to GitHub pages?

Thanks Vimalraj

[krmahadevan]

@juherr

Fyi, the documentation is already served by Github Pages. So you can directly write markdown and the file will be translated.

I was under the impression that in order for github to serve documentation, the changes should be available in a special branch called gh-pages. But that doesn't seem to be the case in our project. I also remember us having to run a hook after a change has been published to the TestNG repository, so that it can get reflected in the testng.org site.

[cbeust]

Julien, so that means, testng.org is mapped to GitHub pages?

Correct: https://github.com/testng-team/testng-team.github.io

Are we talking about just changing the way the doc looks or rewriting the doc? I'm confused.

[krmahadevan]

@cbeust

There are a couple of things that are being discussed here.

1. Structuring of content is being asked by @freakerhere - I believe you asked some follow up questions for which @freakerhere is to be getting back.
2. Look and feel. I got @email2vimalraj involved because we both have been discussing of ways in which we can improve the overall look and feel of the testng documentation portal (Maybe have it revamped to look similar to https://jcommander.org ) and we are bouncing back ideas. The documentation portal to the best of my knowledge is up-to date in terms of content (there may be a miss here and there) but maybe we could relook at how we can help people find information by restructuring it (I personally liked how the jcommander site looks like).

So we are more talking about changing the way the documentation looks like and not necessarily re-writing the doc (which is not needed because the content is valid as of today).

[ayushchatur]

@cbeust

There are a couple of things that are being discussed here.

1. Structuring of content is being asked by @freakerhere - I believe you asked some follow up questions for which @freakerhere is to be getting back.
2. Look and feel. I got @email2vimalraj involved because we both have been discussing of ways in which we can improve the overall look and feel of the testng documentation portal (Maybe have it revamped to look similar to https://jcommander.org ) and we are bouncing back ideas. The documentation portal to the best of my knowledge is up-to date in terms of content (there may be a miss here and there) but maybe we could relook at how we can help people find information by restructuring it (I personally liked how the jcommander site looks like).

So we are more talking about changing the way the documentation looks like and not necessarily re-writing the doc (which is not needed because the content is valid as of today).

@krmahadevan content wise the portal is ok. @cbeust But my major point/feedback are

• To overhaul the look and feel ( major)
• To reciprocate on the content visualization and structuring to enable people find content easily may be incorporate a bit of SEO/Analytics ?

For PR i am very much available but lack experience and need someone to guide and mentor me for a head start.

[yatheeshraju]

hey there ,

i have made read the docs portal with same content as test-ng .here is the link

http://testng-docs.readthedocs.io/

Let me know how i can push the code to this repo .

Thanks, Yatheesh

[krmahadevan]

@cbeust @juherr WDYT ?

[cbeust]

This looks fine to me, but there's a lot of work to carry the existing doc to that format. I'm open to doing this, though, it will modernize the look.

Let's start with the basics: how do you build this doc, @yatheeshraju?

Also, we should probably move this issue to https://github.com/testng-team/testng-team.github.io, which is where the work will be taking place.

[yatheeshraju]

@cbeust i am using https://www.mkdocs.org/ for building markdown(.md) files to the website. is the theme i am using https://squidfunk.github.io/mkdocs-material/ .

Read the Docs does all the heavy lifting http://readthedocs.org/

we push the markdown files to github and Read the Docs builds the docs from github repo .

I am also new to it .only followed some basic tutorial .

i have converted most of the docs to the format .

[krmahadevan]

@cbeust

Also, we should probably move this issue to https://github.com/testng-team/testng-team.github.io, which is where the work will be taking place.

Agreed. But since the conversation has started here, lets use this issue to finalize and then have an issue created in the documentation project once we have decided on the approach. that way the conversation is all in one place.

@yatheeshraju - It looks like readthedocs.org suggests that it would be doing the hosting of the documentation and I also noticed that the URL also changes. So how do we take care of ensuring that we retain the url as testng.org ?

@email2vimalraj - FYI. This is how the entire setup is done for the current TestNG documentation.

[yatheeshraju]

@krmahadevan in readthedocs admin page we can add custom domain which will make the docs to be served frorm testng.org

if we want to use the current testng-team github repo i can build the mkdocs and provide the static files .they can be hosted from github like the current web pages , so will not change the current setup .

[email2vimalraj]

Hello All,

Here are my observations which might help to make decisions:

Gatsby: Gatsby is a framework which helps to build a static site. That means we can build the full-fledged website with documentation page where a typical documentation like site would look like. And moreover, gatsby is built using latest technologies like React and GraphQL. My idea is to build a landing page and docs page. Example of landing page and docs page: https://www.gatsbyjs.org/ - A gatsby website itself.

UI designs will be built using React and docs will be rendered from markdown files. For UI design, the templating is very simple because it is mostly HTML and we easily get contribution from the community as it is JS based. This stack (JAMStack) is very popular now a days.

The gatsby has wide variety of plugins which helps to easily configure the SEO, responsive images, offline support, PWAs.

The build will provide the set of static html files which can be either hosted somewhere or check it in github itself to serve using github pages. Publishing through github pages doesn't change of existing workflow.

Mkdocs: Now the mkdocs on the other hand is built using Python and helps to develop the documentation site. This doesn't support adding landing pages or any other static pages. It uses the jekyll templating engine to convert the templated html to static html. So we have to learn the jekyll templating language in case customization required. The docs are written in markdowns like gatsby. Mostly we should use the existing themes available in the readthedocs, though you can customize the themes provided by readthedocs, it is limited. There are not much community activities in Mkdocs.

The build and publish steps are pretty much similar to gatsby. Readthedocs may not be necessary in case we use github pages.

Now come to some statistics: Trend analysis: https://www.npmtrends.com/gatsby-vs-mkdocs Stack comparison: https://stackshare.io/stackups/mkdocs-vs-gatsbyjs

My Verdict: I go for gatsby mainly because of vibrant community and very active in development. Also, it provides full control to the developer to customize however we want.

Hope this helps to make a decision and move forward.

[drapostolos]

@yatheeshraju Regarding this docs:

http://testng-docs.readthedocs.io/

Is there a way to view the entire documentation in one page? I think this is important for the reasons mentioned here. Second paragraph. I also include the text here for convenience:

For online documentation, make sure that there is a link that brings up the entire documentation in one HTML page (put a note like "monolithic" or "all-in-one" or "single large page" next to the link, so people know that it might take a while to load). This is useful because people often want to search for a specific word or phrase across the entire documentation. Generally, they already know what they're looking for; they just can't remember what section it's in. For such people, nothing is more frustrating than encountering one HTML page for the table of contents, then a different page for the introduction, then a different page for installation instructions, etc. When the pages are broken up like that, their browser's search function is useless. The separate-page style is useful for those who already know what section they need, or who want to read the entire documentation from front to back in sequence. But this is not necessarily the most common way documentation is accessed. Often, someone who is basically familiar with the software is coming back to search for a specific word or phrase, and to fail to provide them with a single, searchable document would only make their lives harder.

[email2vimalraj]

I don’t entirely agree to a single page documentation with given the latest technologies available to overcome the limitation you pointed out. The algolia docsearch could be used here to enable the search across pages, more here: https://community.algolia.com/docsearch/. Please note that this is entirely free for the open source.

And on another note, think of SEO capabilities, if it is a single page, then SEO might not crawl properly with certain keywords. If it is multiple pages with proper titles, then SEO can easily crawl page by page and users can often search directly from search engines.

Please let me know what you think.

Thanks Vimalraj

[yatheeshraju]

@yatheeshraju Regarding this docs:

http://testng-docs.readthedocs.io/

Is there a way to view the entire documentation in one page? I think this is important for the reasons mentioned here. Second paragraph. I also include the text here for convenience:

For online documentation, make sure that there is a link that brings up the entire documentation in one HTML page (put a note like "monolithic" or "all-in-one" or "single large page" next to the link, so people know that it might take a while to load). This is useful because people often want to search for a specific word or phrase across the entire documentation. Generally, they already know what they're looking for; they just can't remember what section it's in. For such people, nothing is more frustrating than encountering one HTML page for the table of contents, then a different page for the introduction, then a different page for installation instructions, etc. When the pages are broken up like that, their browser's search function is useless. The separate-page style is useful for those who already know what section they need, or who want to read the entire documentation from front to back in sequence. But this is not necessarily the most common way documentation is accessed. Often, someone who is basically familiar with the software is coming back to search for a specific word or phrase, and to fail to provide them with a single, searchable document would only make their lives harder.

@drapostolos

No we do not have option to view entire document as one page ,

full docs we have a search option .

[email2vimalraj]

@cbeust @krmahadevan @juherr kindly let know about your thoughts

[krmahadevan]

@yatheeshraju

if we want to use the current testng-team github repo i can build the mkdocs and provide the static files .they can be hosted from github like the current web pages , so will not change the current setup .

Can you please call out the steps that anyone who wants to contribute would need to take ?

@juherr / @cbeust - I am not clear on whether the documentation is being served from testng.org (I remember earlier we used to a php backed web hook url, which we had invoke after we get our documentation changes merged into the GitHub repository, for the documentation to get reflected in testng.org. Has this changed ? )

It would be great if you could please call out the current process that is being followed in order for the documentation to be rendered and also help conclude on what should be our way forward ?

We have two options right now.

1. Backed by readthedocs (suggested by @yatheeshraju )
2. Backed by Gatsby (suggested by @email2vimalraj )

[juherr]

@krmahadevan No more php server since Cedric had some issues with the hoster. So yes, it changed and it is currently backed by Github Pages.

If I understand well, Gatsby can be backed by GithubPages too.

Both solutions look great and more modern than the current one. @cbeust you choose! :)

[yatheeshraju]

@krmahadevan

contributor system setup -Python -mkdocs (pip install mkdocs) -mkdocs-material (pip install mkdocs-material) -clone the repo ( currently https://github.com/yatheeshraju/testng-docs. i can commit it to the official testng-docs repo) -make changes to files (mkdocs serve).To view changes in local system -commit to publish

Admin side one time setup (contributors need not do this) -create account in readthedocs -connect the repo to readthedocs -in advanced settings - -choose single verison -select mkdocs in documentation type -give requirements.txt in requirements file name . -in overview you can build the docs from the connected repo. -in domain settings you can add the testng.org domain to point to it .

I am open to gatsby as well . looks good and customization friendly than mkdocs/readthedocs .

[krmahadevan]

@yatheeshraju - That is awesome detailed information.

@email2vimalraj - Can you please provide the same for Gatsby ?

[email2vimalraj]

@krmahadevan : here you go

Contributor setup:

• Node JS
• Clone the repo
• npm install
• npm run dev => comes with hot reload, so developer can make changes to the code and immediately reloads in the browser

Publisher setup:

• Add / update the markdown files with documentation content
• npm run build => generates static HTML files
• Can publish the build directory through github pages or any hosting provider or even through a CDN.

Please let me know if you need further details.

[cbeust]

Personally, not a fan of forcing contributors to have to install Node, this is going to discourage a lot of people from contributing (not that the TestNG docs change much, but still).

I'd rather have a doc portal where contributors can just clone the team-testng.github.io repo, edit a few files, commit and push them, and the CI/CD pipeline takes care of building the docs.

[email2vimalraj]

+Update+: Added more detailed steps of build and deploy process.

@cbeust : I should have been little clear. For the doc site, we’ve two different types of contributors,

• UI contributor:

  • Nodejs is required
  • Fork & clone the repo
  • Make react based code changes
  • Commit and raise a PR
  • Reviewer reviews and merge the PR to master
  • The CI/CD process will initiate which will build and deploy (refer the CI/CD steps below)

• Doc contributor:

  • Fork & Clone the repo
  • Add / update the markdown documents
  • Commit and raise a PR
  • Reviewer reviews and merge the PR to master
  • The CI/CD process will initiate which will build and deploy (refer the CI/CD steps below)

• CI / CD workflow:

  • Obtain the github access token for the travis CI
  • In Travis, for the repository, add a new secret environment variable GH_TOKEN with the access token as a value
  • Make sure to not to toggle the display in build logs setting as the token is best to remain secret.
  • There will be a deploy script in the package.json which will build and push the public folder (this folder will contain static htmls) to the gh-pages branch, since we wanted to serve from the github pages.

"scripts": {
  "deploy": "gatsby build --prefix-paths && gh-pages -d public -r https://$GH_TOKEN@github.com/cbeust/testng-docs.git"
}

Please notice the GH_TOKEN environment variable which will be replaced at run-time.

The sample .travis.yml will look like as below. Travis will trigger the build only when a commit is pushed to master branch:

language: node_js
before_script:
  - npm install -g gatsby
node_js:
  - "10"
deploy:
  provider: script
  script: cd docs/ && npm install && npm run deploy
  skip_cleanup: true
  on:
    branch: master

Hope this is clear now.

[krmahadevan]

@cbeust - Can we please decide on this ?

[ayushchatur]

@vimal: this means that look and feel will be determined by the react code and text/content will be taken care from docs.

But i am still in lingo as to how the two will be coupled? For example let say i want to add a new section to the documentation for a new feature in that case do i have to work on both parts? or that can be taken care by react. If yes, then we have to discuss on the architecture of the react framework. it would be helpful if you can create a demo. I am open for help

[email2vimalraj]

If the new section involves some UI changes like adding new styles of look and feel, then yes it requires some react code. If the new section is just a doc then adding or updating the markdown is pretty much what we need to do.

For example, you can refer my blog site which I’m working on now: https://github.com/email2vimalraj/site. I add my markdown in posts folder and every time I push to master, the site will be deployed automatically to zeit now - my CDN provider and accessible via https://vimalrajselvam.com with zero downtime.

This is a typical way for most of the blogs or doc sites backed by Gatsby. I believe Krishnan’s blog also works in the same way.

Please let me know if you have any questions.

Thanks Vimalraj

On Sun, 19 Jan 2020 at 9:02 PM, ayush notifications@github.com wrote:

@vimal https://github.com/vimal: this means that look and feel will be determined by the react code and text/content will be taken care from docs.

But i am still in lingo as to how the two will be coupled? For example let say i want to add a new section to the documentation for a new feature in that case do i have to work on both parts? or that can be taken care by react. If yes, then we have to discuss on the architecture of the react framework. it would be helpful if you can create a demo. I am open for help

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cbeust/testng/issues/2219?email_source=notifications&email_token=AAJIRXWNWAO2AS2VXOVBZ2DQ6RFPTA5CNFSM4KC4GERKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKRTMY#issuecomment-576002483, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJIRXS7FJQZ7XQNRGDRL7LQ6RFPTANCNFSM4KC4GERA .

-- Thanks, Vimalraj

[email2vimalraj]

@cbeust : Need your input to go ahead.

[cbeust]

I already stated my concern previously.

I'd prefer the docs to be editable by just cloning the documentation site, editing a file, and committing. Requiring to install Node concerns me.

[sailfishdev]

Great discussion! Thanks for starting this thread and posting so much of useful information 👍.

I also wanted to share my two cents to change the look & feel of the testng official website which I believe will attract the community to be involved in this wonderful project more than ever 😃.

I have created a basic PoC to present my case here - https://s4ik4t.github.io/

This website is built on Vuepress and deployed via Travis-CI/GitHub Pages automated pipeline (using markdown). Yes, Vuepress has a dependency on nodejs however (as per my experience) for vast majority of the contributors (I'll say 98% 😉) that should not make any difference because they can easily edit the markdown page directly from GitHub without requiring anything else (apart from Pull request approval 😄).

There are other useful customization options (e.g. inbuilt search, SEO etc.) which would be one time activity and I will be happy to contribute to make this happen. Vuepress is a single page application and is built keeping technical documentation in mind.

I would love to know your thoughts regarding this @cbeust @krmahadevan @juherr @email2vimalraj @yatheeshraju @freakerhere

Thanks for testng! Keep rocking and stay safe!

[juherr]

@cbeust Any update? https://s4ik4t.github.io/ looks very nice.

[the-code-journal]

I personally really liked the approach for @yatheeshraju . Mkdocs really keeps it simple with just two libraries - mkdocs and mkdocs-material. Plus, it provides out of the box full text search feature.

Also, the edit feature is directly available on the website for anybody who wants to make edits to the documentation site. It opens the github file for editing and on submitting the same, it creates a PR for review. For anybody contributing to just the documentation, they don't need to install mkdocs. Everything can be done via the browser alone.

[krmahadevan]

Closing this issue. The documentation portal now uses "AsciiDoctor" for managing the documentation. Chose this so that

• Our documentation content does not rely on any other service except for github to host it.
• The contributions to documentation is merely in the form of .adoc files.
• The process of generating the html files just boils down to using maven for doing the translation and does not require any additional work.

All said and done, I am seeking help on this issue which I created that aims at making the table of contents dynamic.

Any help with this issue, would be greatly appreciated.

Issue: https://github.com/testng-team/testng-team.github.io/issues/50