Overview modules

[milachae]

Make an overview of all available modules on the different clusters.

• [x] Automatically generate markdown table with all the available modules based on the lmod output #552
  • [x] fetch data from lmod
  • [x] convert to mk table
• [x] Change to dynamic searchable datatable #554
  • [x] put the data in a JSON file
  • [x] Load data and populate table
• [x] Add Overview for each software #559
  • [x] Table with different versions on the different clusters (toolchains)

[boegel]

Just so we have a record of this: we already have a JSON file that is automatically updated (along with the Lmod cache), which provides most information that is necessary to create a software overview, it can be obtained from /apps/gent/lmodcache/modulemap.json when logged in to the HPC-UGent login nodes. Information missing there is the description + homepage that you get for individual modules via module show which would be nice to include in the software-specific overview of available versions.

[stdweird]

https://github.com/hpcugent/vsc-modules/blob/master/lib/vsc/modules/cache.py is where the magic happens. i prefer to have a single metadata (incl homepage description) for each module (somehow guess what the latest is and use that?)

it's just to keep the size of the json low.

[stdweird]

also for the record ;)

if you want more interactive access to this, you can look into https://osc.github.io/ood-documentation/latest/tutorials/tutorials-passenger-apps.html: you can then create a ruby/passenger app that has access to whatever you need. not sure how this overview would like in the docs, but it will be huge and thus slow if shipped in some static form.

also of interest might be https://github.com/hpcugent/vsc-accountpage/pull/1367 where we imagined to use the django build in form searches to do the magic. (it's not the same as this, but has the same problems wrt size and speed)

[boegel]

@stdweird Do you prefer only having a software overview available in the web portal as opposed to in the docs (which are public & hosted on GitHub)? I mean outside of forms to create a desktop apps & such, so a more "static" table overview (but which is somewhat searchable, like the table-with-filter approach that @milachae has a proof-of-concept of), which eventually also includes extensions (like numpy which is included in the SciPy-bundle module, R packages, etc.).

Shipping all necessary data to produce an overview of installed modules isn't that huge, the page still loads quickly enough in practice (@milachae has been experimenting with that).

[stdweird]

@boegel just pointing out some options. if it's fast enough and does what you want, go for it

[milachae]

Some ideas:

1. add the whatis information to the page (watch out, ml show can take some time when executed from python)
2. add a system to attach markdown files to the currently auto generated files