Documentation improvement - RawPedia replacement

Since https://discuss.pixls.us/t/an-update-on-rawpedia/34168/43 often falls by the wayside and people forget about it, perhaps it might be beneficial to track the issue here.

A quick summary of the issue(s):

MediaWiki has maintainability issues that seem to be getting worse, and the person primarily responsible for administering it has generally moved to static site generators for all of their other projects and recommends that the RT team do so also
The requirement to have an account approved in order to submit content is a barrier to people being able to contribute to documentation

In general it seemed like the response to the concept was positive for all involved

The proposed solution was hugo backed by a gitlab or github repository, which has the following advantages:

Anyone with a github account can submit a documentation change, but it must be approved by someone with merge rights to the documentation repository. Handles the anti-spam rationale for RawPedia's current access control, while making it drastically easier for drive-by documentation submitters to submit requests
Hugo seems to be heavily relied upon for documentation by many projects, including darktable whose dtdocs infrastructure we can use for inspiration
An SSG that is backed by github and outputs to a static website using Github Actions is going to be VASTLY more secure than MediaWiki

Rough task list would be:

Get a dump of the current state of Rawpedia for experimentation with one of the conversion processes. https://gitlab.apertis.org/docs/mediawiki-to-hugo-conversion appears to be a bit more robust and require less low-level access to the mediawiki server backend than https://github.com/xenlo/mediawiki-2-hugo - I'm assuming this would be patdavid or paperdigits (not sure what their github usernames are, I need to look that up...)
Get a roughly working version of a migrated installation in a test repository - I can start with a personal repo and then migrate it to an organizational namespace later
Also something that has been discussed and seemed to be received favorably by most parties is to move all RawTherapee related repositories to a Github organization namespace instead of @Beep6581 's personal namespace, which makes the project more robust to instances of leadership burnout. I think having an organization repository will be far more important when the documentation is also hosted in a Github repo
Create github actions that run hugo and publish to the appropriate server (currently Pixls infrastructure I think, and I would assume it will remain that way?)

Other than the "get dump from server" tasks and "move to organizational namespace" tasks, I'm volunteering to start working on this. I've spent some time learning hugo already and really need to learn Github-action-foo, and I'm finally mostly recovered from the mental health issues that had me not really work on anything RT-related (or frankly much else) for over a year. (Hopefully I'll have some treatment for suspected but as-yet-undiagnosed ADHD in a bit over a week, which should also help significantly with me stopping the procrastinating and actually DOING something. :) )

Issues that need to have solutions discussed (some of this came up in the linked Pixls discussion):

Handling of translators/translations of the documentation - I should look at how dtdocs handles other languages
Hosting of images and other media. Github's LFS support has SEVERE limitations before you have to start coughing up money.

I've been learning a bit of how Hugo works, and I wish to share some thoughts about it and websites in general.

From what I understand the idea is that a user should only worry about writing markdown, and a bunch of parameters in the frontmatter (Hugo's file metadata).

Given that and the suggestion to not use plain HTML within markdown, next are some points to think about.

Whenever you read «must», «should», «have to» or the likes, read all of them as suggestions, as those are only my personal point of view about them.

1- About Hugo:

they say is easy on the user, on the people writing content... Well, truth is it is and it isn't: it all depends on the efforts made by the person who has built the theme used in the website
there are things that are just there in mediawiki (special pages, galleries, online wysiwyg editor, ...) that are not possible, or at leat not easy to code in Hugo
at the time of creating a theme, it must be crystal clear what is needed and how it should be shown on the rendered pages. Hugo may get really convoluted depending on the needs of the site. However I wouldn't expect such scenarios in a site for RawPedia
Hugo can build any type of website, as long as it is static. That suggests that maybe it could be a good idea putting it all together, the main website and the documentation, as the main website is also static

2- About the website users, a.k.a. the page editors:

as they are the ones creating content, they must concentrate on content, not in some site maintenance tasks: let's say somebody not necessarily knowledgeable about Hugo internals writes a new content page and submits a PR. That person shouldn't edit the site configuration, ever, but that leads to a site maintainer having to write a new menu entry linking to that page in the site config. Shouldn't it be automated creating the menu entry some other way?
they are left with markdown, which is not ideal to add content to usual websites. Markdown is very limited regarding styling content. Which leaves us with shortcodes (some sort of macros), and somehow users should be aware of the existence of such shortcodes and how to use them
should they create content in page bundles (a folder where everything involved with that page is there, including such page), or pages pointing to a shared folder where the resources are located? E.g. all the images used in the site. Before answering, consider what happens with page bundles mixed with translations... Do you clone all the resources in the translated page bundle?

3- About content:

images: a photography site must contain pictures, doesn't it?
images (another take): they must have captions, it really is a must, and markdown is not up for the work (you can't add captions with just markdown)
images (final take): what if you need to compare 2 images? How do you create a gallery, or a slideshow of images, or a sliding comparison (where one image is on top of another and you move a slider one side or another to view one image or another)?
tables: markdown is not really good to create tables. To be precise, it's really not good at all to create responsive tables. Responsive tables are a really difficult thing in websites, specially when rendered on smartphones, but they are used in RawPedia and there are really demanding needs for them (such as sorting columns)
internationalization (i18n): add this and the theme becomes more complex to code. But the good news is it seems that there is no need to create a page in the main/default language before posting it in another language
how is it planned to manage the longest articles in a way that people don't get scared of when realizing the amount of text they have to read...? Do you add a TL;DR box at the top of the page, where you can add a summary with the main points of the page? That is, if you're in a hurry, just read that box and you will know how to proceed with RT. Or maybe you create a pager and split content in multiple subpages?

4- About the website itself:

to bootstrap, or not to bootstrap? Do you wish to be constrained by a framework? Bootstrap is easy to quickly start building a website, but when you begin to have specific needs (not covered by the framework), it really is a hindrance
toc: there are multiple ways of adding a page table of contents. I think the most desired one is such a toc that highlights the section of the page you're currently reading
menu: how do you create a good menu, pointing to aaaaall the tools available in RawTherapee? One which doesn't force you to keep scrolling on smarphones displays? One that doesn't force you to edit the site configuration everytime a new article is uploaded? One that also has links to home, search, language selection, color scheme, ...?
color schemes: it is expected on modern websites to have at the very least a dark and a light color scheme. Much better if there's a 3rd color scheme to choose from
search feature: do you implement a local search engine, or will rely on google's one?
accesibility: well, this one is quite quite tricky, as you need people using real accesibility devices (screen readers, ...) to tell you if you are doing things right. And nowadays it seems it is a must creating an accesible website
SEO: this is highly technical, but needed. There are ready made solutions out there to be included in a Hugo project
sitemap: every site must have one
social networks links: it is expected to have at the very least one (github), but people may have more. E.g.: it wouldn't be a bad idea having one link to the project github url, and another one pointing to the RT section of pixls.us
latest modified or added articles listed in the homepage: which will most probably be the pages people are seeking the most, so they should find them quickly
do you plan to allow some form of comments? That requires an external tool (staticman, ...)
which structure you would like for the website? How and where should the navigation bar, content, footer, etc. be rendered? How should it morph when rendered in smaller or bigger screens (a.k.a. responsive site)?

There is a good amount of decissions to be made when building a theme, which will constrain the way the website is managed. A theme developer may decide how everything should work, but afterwards everybody should feel happy with it: site maintainers have little work to do, content creators have a relatively easy task with markdown, and the site readers have a smooth experience with the site.

As a last note, the easier the content can be created, and the easier the site can be maintained, the more work for the theme creator because more and more things must be coded to automate features. So, if you want an easy experience with the site, with Hugo you have to carefully craft the theme.

@patdavid FYI

That person shouldn't edit the site configuration, ever, but that leads to a site maintainer having to write a new menu entry linking to that page in the site config. Shouldn't it be automated creating the menu entry some other way?

The website automatically adds news articles and download links, so I assume it will be possible to automate it for RawPedia.

they are left with markdown, which is not ideal to add content to usual websites.

I see that MediaWiki markup has image captions. What else is missing from markdown that markup has?

The website automatically adds news articles and download links

I'm not sure I understand this. Do you mean the current RawPedia website? If so, the links present in the homepage, those that may be more or less felt as a menu, are not automatically generated, if I remember correctly. They have to be typed by hand, instead.

In a Hugo website, it is indeed possible to automatically generate links to new pages, but how they will be shown depends on how the menu is designed. I've managed to automatically create a menu with all the links (pages) present in a website, but it quickly gets huge if you show the links to all the pages of the site.

In my opinion, the menu design should be carefully designed to show the essential links while still giving a good user experience (too many links means too many page scrolls to find the desired link, and that's bad user experience).

What else is missing from markdown that markup has?

As a reference, take a look at this Hugo thread. (As a side note, jmooring is a main contributor). There, it is clearly stated that markdown only has provision for a title tag, which is rendered as the image tooltip. There is no way to add image captions with plain markdown.

This is the way mediawiki encodes images: [[File:filename.extension|options|caption]]

And this is how Hugo encodes images: ![alt text](/path/to/image.extension "title")

The reason why I say images should have captions is explained in this website (among others).

At RawPedia, most of the images show an example of what's being explained in the text, so it makes sense (if it's not a real need) some caption explaining why that image is linked to the text.

The usual approach, as far as I've seen, is using shortcodes, either to add custom addons to an image, or to use the figure html tag. It's not that difficult, there are lots of examples out there, but the user writing content should be aware of the existence of such shortcodes, somehow.

If needed I can save a webpage (html file) showing what can be done with plain markdown and share it here. Not too fancy styling though, and it's written in Spanish... (Sorry, no live website available).

It looks like it may be possible to implement captioning in Hugo - https://sebastiandedeyne.com/captioned-images-with-markdown-render-hooks-in-hugo/ is one possible approach.

Obviously there is going to be a lot of experimentation involved in this process. Ideally we'd get a dump (via @patdavid ? - not sure why attempting to @ him doesn't autocomplete properly... I only see patdavid-test...) to fiddle with as an experiment.

@TechXavAL I'm referring to the RawTherapee website (www.rawtherapee.com). The News and Downloads pages are automatically generated from the articles in the news and downloads directories. In this merge request, I added a news article on the updated macOS build. Note that there are only two files: the new news article and some edits to the 5.10 download page. I did not need to manually add anything to www.rawtherapee.com/news/.

@Lawrence37

Ok. Sorry I didn't get it. I thought we were talking about mediawiki all the time... And yes, that's a Hugo feature when integrated with git (github, gitlab, ...).

Now let me distinguish between several concepts:

the news page you are referring to is a list of the latest updates or uploads of certain kind of pages (e.g. pages of type news), sorted from latest to oldest
the news link you click to go to the previous page is located in the nav bar, and it's hardcoded. It's not automatically generated (it somehow could be, but it requires certain folder organization)
there's no standard menu in the RawTherapee website. At least no the usual dropdown menu, which is what I was talking about
usually most of the links to get to any place in the website are located in the menu+nav bar. If some link is not there, then it's difficult to arrive to some place in the website. E.g.: if the news link was located inside the homepage (somewhere in the middle of the text), then anyone coming to the website directly to e.g. the download page, would not have any clue that the news page exists. They would only know it if going to the homepage and reading the text to finally find the news link. Convoluted, at the very least.

So, to rephrase my previous comments about the menu:

the nav bar should only have a few items, and they could be hardcoded because they should never change, no matter where you are in the website (link examples: home, search, color scheme, language selector, articles, RT download page)
the dropdown menu may be a vertical list of links, like we are used to see in many desktop applications, or a group of links to pages where there are more options, or a mega menu
most probably a list of links to all the pages included in RawPedia won't be a good idea, because there are too many
no matter where you create the links to all pages of the website, those are usually not automatically generated. They could be, but it's not straightforward, as you need some folder+archive organization if you want to group pages depending on some requirement

@Entropy512

Yes, that's one approach, but let's not put the cart before the horse. The requirements, what is needed about images must be defined prior to seeking for solutions. Same strategy for everything in the website.

In my opinion that's a quick and easy solution for captions, but is hacky and doesn't solve all problems. You replace one tag for another, and in the end you are doing mostly the same work as you would with a shortcode.

In everyday content creation you need to add an image, an explanation of what you see in the image, the author of the image, the source url of the image, and the original licensing of the image. You know, it's a must, because that's what is currently required at RawPedia (well, maybe the explanation is not required). And that falls completely outside markdown capabilities.

By the way, with that solution all the images in RawPedia will look the same, and will link to the same image. That is, for large displays you need a large, heavy image, and for small displays like smartphones you will download the same large image. And you won't be able to use css class selectors to change the position, appeareance or size of the image. That's fine if that's what you want. What I say is that you should be clear about what you want or need prior to code anything.

The requirements, what is needed about images must be defined prior to seeking for solutions.

Writing a requirements document without a thorough understanding of the potential solution space is a paper tiger - a waste of time because you wind up writing a requirements document that cannot be met by any viable solution. So you wind up throwing away a bunch of your requirements because they turned out to be unrealistic. Our solution space is already limited - must be free and open source, must be mostly off the shelf with low amounts of customization (no touching core code, only data and ancillary scripts such as JavaScript macros, etc.)

As to the worries about whether a rawpedia replacement is mobile-friendly - given that rawpedia is the document for an application which has no support whatsoever for mobile devices, I really don't think there's that much benefit in worrying about mobile devices. Anyone using the software is going to have a full desktop browser at their disposal. Obviously mobile friendliness is a nice-to-have here if it can be achieved without significant compromises but I really don't see smartphone friendliness being a hard requirement for the documentation site for something that is not going to ever run on a smartphone or even a tablet in the foreseeable future. At least the Hugo theme I've poked at in the past (ReLearn) handles mobile layout well, with the only limitation being no mobile-specific image resizing. Not like Rawpedia handles that well - it serves a smaller thumbnail (300 pixels wide) to my desktop for images on the Exposure page than to my phone (600 pixels wide)

Note that right now my focus is on rawpedia. The website might have its own issues (and THAT is a case where mobile friendliness is more likely to be of benefit) but that should be kept to a separate discussion to avoid confusion. Perhaps in the future if both are using Hugo there will be a way to integrate the two, although so far looking over in darktable-land, that hasn't happened. I do agree that the lack of images in dtdocs is something that we don't want to follow, but I think there are plenty of solutions to that issue.

My biggest concern still is the issue of image storage - LFS and github are not a good mix. Linking to external image hosting is theoretically possible but that winds up with some of the same challenges rawpedia has (access management, etc). Not sure if the solution might be a self-hosted git repository on pixls infrastructure? LFS itself isn't bad, the problem is Github's pricing for LFS. I wouldn't be as concerned about the 1GB limit except that it seems like every single clone of the repo gets counted against your transfer (BAD for a public FOSS project) and any image that gets removed/replace remains in LFS as part of history.

It seems to me that there's always a middle point..., but anyway.

Just to say things quickly:

an already existing theme with almost everything needed already there is desirable
there's no need to serve mobile users
there's no need to merge RawPedia and RawTherapee websites (maybe in a future)
images are necessary
shortcodes may or may not be desirable

For image storage, maybe coolify could be a solution for self-managed, open-source, hosting server?

Yup, those look good to me. I'm open to suggestions on documentation-specific themes, but it's my opinion that "ReLearn" seems to be the closest in layout to MediaWiki of the options in https://themes.gohugo.io/tags/docs/ but maybe I missed another option. I think there's definitely value in fiddling with a few options as experiments first.

I'll leave judgement on coolify to @patdavid since that might be redundant compared to a lot of the stuff he's already set up. One of the big things I want to avoid, if possible, is duplicating one of the big flaws in mediawiki which is the risk of spambots clogging the system if we remove the barriers to account creation.

I've thought about coolify as a cdn: it serves media to github pages, so you still have free CI/CD github services, account management, known server environment (github)... But as I haven't spent time figuring out what's possible with it, I will let the pros to decide if it's good or not.

Now, here is another option for images: Hugo Easy Gallery, with a live sample page.

It is a shortcode that you include within the markdown file, specifying all needed information. It's a bit more typing work than just including an image, but I think it's worth it. And if it finally is acceptable, it could be customized to include all the must have information (source url, original author, licensing).

The same shortcode can be used to add standalone images and galleries. It's all explained in the link.

My comments will be limited. On the one hand, I have a very poor command of English and, on the other, I have a minimal command of Github and all these tools. I copy what I'm told.

I trust the team to find a solution (or solutions) that will be comprehensible and transparent (progressive?) for the majority of users and developers. These major transformations of our media must be accompanied by good communication.

As a reminder, I've written numerous articles for Rawpedia, notably the "Local adjustments" section in collaboration with Wayne. It's a big (very big) job.

I would add: 1) "je ne botte pas en touche ".. I don't take sides, as we say in French, 2) I'm ready to cooperate 3) it's a good idea for the system to focus not just on the "how", but also on the "why", with the notion of Retour d'expérience (REX) as a source of learning.

Jacques

So updates:

Now that I'm properly medicated for ADHD (WOW what a MASSIVE difference) I'm going through all of my previously-partially-started-but-never finished projects. I definitely have some time, motivation, and focus to start poking at some proofs of concept. Biggest thing I'd need would be a dump of the current wiki's database as described in https://gitlab.apertis.org/docs/mediawiki-to-hugo-conversion
Still need to figure out how to handle images. It's a tradeoff between storage cost and ease of use - git LFS would be the easiest to manage, but github REALLY likes to charge you for it. (Although I need to do some experiments, because it turns out Github doesn't apply a LOT of usage limits to public repositories so maybe this concern is moot, although https://github.com/orgs/community/discussions/62051 implies otherwise. ORAS ( https://oras.land/ ) combined with Github Container Registry is free, but a lot more difficult to manage since the images are in a separate location that isn't version-controlled.

Edit: So far, it looks like for a public repo, it counts against your storage quota but not against your bandwidth quota. I've pulled from a test repo multiple times and my bandwidth is still showing 0, but maybe there's some latency here. As long as the repo is garbage-collected occasionally, it might be OK if it continues that pulling from a public repo doesn't count against the bandwidth quota.

I was looking at the CLI just now and almost got tripped by the documentation on Rawpedia (I was looking for a 16 bit float output.. had to read the code as the CLI doc doesn't mention -b32 or -b16f ) and was looking for a contribution pathway which led to me finding this issue.

While transition is being investigated is there scope to split off any work to set up a base contrib pathway? For the CLI flags, as an example, I'm happy to spend a bit of time writing up docs based on main-cli.cc and submit them as patches for mediawiki but I don't know if that's the appropriate route as that's still review work and pollutes any transition effort.

If it is, i can self-make an issue to draft up a "doc contribution" issue to draft some words etc etc and try carry that but I'm aware that docs are work, and a time commit.

You can always take a look at the Spanish documentation and see if the explanations are good enough. Then translate it yourself and maybe upload the English version here opening a new issue.

Beep6581 / RawTherapee

Documentation improvement - RawPedia replacement #6999