Suggestion: migrate user documentation to this repo

[cmeessen]

We would like to suggest to migrate research-software-directory/documentation to this repository.

We think that this could have some benefits, e.g.

• the RSD repo always comes with the current user documentation
• keeping the user documentation in sync with the RSD might become easier since no additional pull request has to be created if we want to update the documentation together with a feature update/bug fix

Apart from this, there are several issues updating the dependencies of the documentation. But this might have to go into a new issue.

[dmijatovic]

@cmeessen @ctwhome @jmaassen @ewan-escience @fembau

Yes we need to update documentation library. There is a reason why the user documentation is in the production repo. For documentation websites we use the github pages as hosting platform. The github pages of the main RSD repo host the developers documentation.

I think we need to discuss and decide about :

• What package/lib we should use for documentation? The current one is not actively maintained anymore. I see lot of projects using docusaurus but I don't have experience with it. We could also use SvelteKit or Astro(?). In all cases the idea is similar, having markdown pages being served as website. We could use documentation pages to test new technologies and get experience with these (just an idea).
• Where the documentation will reside and be hosted? The current user documentation is in our production repo as it uses gihub pages as hosting platform. We can move it into main repo and than maybe have one documentation "project" containing both user and developers documentation?

I am interested to hear what are your thoughts about documentation approach.

[jmaassen]

Many of the projects at the eScience Center use sphinx for documentation. Some examples (there are many more):

• Kernel Tuner
• MUSCLE3
• ESMValTool
• Parcels
• LitStudy
• MatchMS
• Spec2Vec

Many of the well known python projects also use it:

• jupyter
• flask
• numpy

While originally designed for python documentation, it is now widely used for other purposes. It it also capable of generating other formats than HTML (pdf, epub, etc.)

It's very suitable for both user and developer documentation.

edit: According to Ewan, sphinx can be a bit complicated to use initially. Also, the default format is not markdown but reStructuredText which has a different syntax

[ctwhome]

To ensure low development maintenance, the documentation has a GitHub action that triggers automatically. However, it may be more convenient to combine repositories to simplify the pull request process. We all know that maintaining documentation can be challenging, so any efforts to reduce development fatigue would be beneficial.

Docusaurus might be a better choice since it is based on React. Moreover, Netlify recently acquired Gatsby, which is also based on React. There are, of course, other systems available, which you can check out at https://jamstack.org/.

You can use Svelte with MDX via https://madewithsvelte.com/mdsvex, but it will require custom code. Therefore, I recommend selecting a platform that is specifically designed for documentation, such as Docusaurus, VitePress, etc.

[jmaassen]

Jekyll is another option we frequently use. For example:

• Amuse
• CFF
• Xenon

The integration with github is excellent, as the github pages are based on Jekyll. Jekyll seems to focus a bit more on ease of use than Sphinx. As a result there are 1M+ github.io pages based on this.

It also has a huge number of themes available, both free and commercial (see https://jekyllthemes.io/free), and some focus on developing documentation such as:

• Just the docs
• Documentation
• Minimal Mistakes
• Docs (commercial)

and many others.

It is based on markdown, it shouldn't be hard to re-use the existing documentation.

[ewan-escience]

I'd prefer something well established and stable, and both Jekyll and Hugo seem to fit these criteria.

[ctwhome]

It's important to consider if there's a genuine problem or requirement with the current solution, and to identify its limitations beyond being in maintenance mode. If we introduce a new platform, it will bring about changes to the style and configuration. I propose that we avoid changing frameworks until version 2 is released. This will help to alleviate some of the development pressure.

[dmijatovic]

Thanks for your reactions and great ideas! I see three main areas we need to decide on. Let's discuss it during the standup.

1. Where the source code of user and developer documentation will be stored. Now the
   • user documention has own repo
   • technical/developer documentation is in the main rsd repo
2. Where the compiled user and developer documentation will be hosted? Now both documentation are hosted on github pages in the separate repos and have separate urls:
   • user documentation
   • developer documentation
3. Which technology to use to generate documentation website(s)? Here the core requirement is to use md as raw format so the documentation can be created/edited by "less-technical" person(s). The md is compiled to static html website. Currently used solution vuepress is not in active development anymore. I expect that we will need to migrate to something else at certain point (longer term). I do not expect we are able to migrate to new tool before first v2 release, but we can start discussing various options and decide between proposed solutions: Docusaurus (React/Node), Vitepress (Vue/Node), Hugo (Golang) and Jekyll (Ruby). If I would need to perform migration I would use Docusaurus. It is specifically designed for documentation websites (Vitepress too) and widely used. I would need to spent some time to read the documentation but because it is React/Node stack I expect to be able to learn it fairly quickly.

Searching on the web about the documentation tools and approaches I saw the idea to keep the documentation source code in the same repo as app source code and publish (push) compiled documentation (static html) to a separate guthub repo and there use github pages (github pages use in this case the main branch). I am advocating for that approach. BTW, I saw some examples with github action specifically made for github pages. I would need to further investigate this approach. The CI approach I would like to propose is that raw md is in rsd main repo and on change in that folder the action would start, build and publish to separate repo that uses guthub pages. It would be fully automated process after PR is merged into the main branch. The current publishing process is also automated afaik.

[jmaassen]

For some previous projects I used a combination of Jekyll, markdown files, generated documentation and github.io, all in the source repo. This was very simple to use.

I would very much prefer to go for something extremely stable and widely used that integrates well will github and our build pipeline. Jekyll and Sphinx seem to be the obvious choices then.

The advantage of Jekyll will be that the migration will be almost trivial, provided we don't mind that the design of the documentation pages change a bit.

[dmijatovic]

After today's discussion we agreed on the following approach. Not everyone participated today and we would like to hear your feedback.

Short term

For the initial v2 release we use the current approach. We simply check our current documentation and update the content if needed. We assume that documentation can be build and deployed without any problems.

Long(er) term

We want to migrate to new documentation tool. There are 3 candidates we want to evaluate: Docusaurus, Hugo and Jekyll. We do not have experience with any of them. Therefore POC's should be done in order to decide which tool suits our needs and we can optimally work with. The POC's should evaluate the tool on each of the requirements listed bellow. We can use grade 1-10 maybe?

Requirements for new documentation tool

The requirements are listed in order of importance

• Markdown pages. It should be easy to migrate the current documentation MD files into new solution.
• Should work with our licensing engine: the licenses headers are added at the top of each file in our repo.
• Version support. The tool should support documentation for different versions of RSD. For example v1 and v2 documentation pages.
• Multi-language support. The tool should support documentation in different languages. We start with English but we expect German, French and Dutch (at least) to be added over time.
• Search option. Having good search option would be improvement compared to our current implementation. Some solutions offer integration with Algolia for full text search of documenation. Other approaches are welcome as long as the search adds value to user experience :-)
• Docker integration. It should be possible to run the documentation in the Docker container without need for installing additional software on local machine.
• Easy to publish / CI integration. The documentation should be easy to publish to any webserver. We think that static html is most convenient format. Github action / CI approach for generating the output is preferred. Note! Support for versioning of the documentation implies that documentation of an specific version is published at the same time as the version of application.
• Visually appealing and consistent with the RSD app styling. The RSD aims to be attractive and easy to use tool for our users. The documentation should have similar aspirations.

Location of documentation

We decided to combine user and technical documentation into one documenation website. The source code will be in the main RSD repo. We have not decided where documentation will exactly be published. We are considering two options:

• Use current documentation repo and github pages. In this case the url of user documentation remains unchanged: https://research-software-directory.github.io/documentation/
• Host the documentaiton on one of our VM's. In this case the documentation url's will be changed to new, to determine, url.

@cmeessen @ctwhome @jmaassen @ewan-escience @fembau This is summary of our meeting about the documentation. Please react if you have any additional suggestions objectsions or advice.

[dmijatovic]

Docusaurus evaluation

The Documentation is good. I also found videos listed below quite useful.

• Navigation bar and versioning
• Translations
• Local search
• Algolia search

Remarks

• It is not possible to use licensing header and front matter definitions at the same time in the markdown file. Page title, menu label and position could be defined in the filename too but I think that using separate licensing file and front matter definitions (title, description, menu label, menu position etc.) in the header of markdown file is the optimal approach. @cmeessen suggested alternative approach for licensing. I think additional license file does not produce lot of overhead but let see what others think..

• Local search supports multiple languages. Note! That search is done over all versions. This will produce results found in any version (see image below).

• Publishing of documention as static html to a separate github repo can be done with build in script npm run deploy

Source code

Note! I tested docusaurus features using my private github account. These repo's are public:

• source code
• published version repo
• published version

@cmeessen, @jmaassen, @ctwhome, @ewan-escience, @fembau Anyone wants to try Hugo or Jekyll on these points?

[dmijatovic]

After some thinking and discussions we decided on the following:

• we will migrate user documentation from separate repo to the main RSD repo into documentation folder. The documentation will cover users and developers documentation. Currently we have only developers documentation in the main repo.
• we will use Docusaurus for this purpose, using basic npm (no yarn/pnpm or other). Docusaurus is optimized for documentation and seem to be a good fit for what we need.
• we will use local search as tested previously (see my previous comments in this issue)
• we will use separate license files (for each md file we create license file)
• we will not use versioning feature of Docusaurus, instead the documentation will become part of RSD release cycle. The release will also include documentation image. The documentation versioning will be covered by RSD versioning. For this purpose release Github action need to be extended.
• the documentation will become part of RSD deployment cycle. The documentation will be included in the docker-compose and will be hosted under /documentation route of main app. For this purpose docker-compose file should be extended.
• we will start with only English version. Helmholtz is welcome to contribute German version if they wish. Docusaurus has i18n support we can use if needed.

@ctwhome, @ewan-escience, @jmaassen, @cmeessen, @fembau If no objections to this proposal I will start working on the implementation next week.