Extract site-creation library from ocaml.org repo

[bluddy]

People have put a lot of effort into ocaml.org, and I think a big chunk of that has been engineering effort to make the template engine work. Perhaps it would be useful to spin out the markdown-to-html part of ocaml.org. Clearly there's a niche for this kind of tool (see docusaurus). It would also help clarify the API, document the library, and perhaps help add features.

Thoughts?

[pmetzger]

I don't understand the ocaml.org build process yet, but I'm generally in favor of factoring out useful tools. I suspect only a few of us are actually available to work on this, so if it is feasible, you might want to look in to what would be involved and maybe do the work?

[agarwal]

The "markdown-to-html" functionality is already externalized. It is done by omd. The templating stuff is done by mpp. The code within ocaml.org is basically a complicated build script to call these tools in a certain order. There is some stuff unique to us, e.g. calling out to ocaml to auto-evaluate code blocks in the 99 Problems page.

To be honest, I think we should stop using both omd and mpp. We're not the experts in static site generation. There must be other industry standard tools that we can leverage.

[pmetzger]

There are a lot of static site generators out there (say Jeckyll) though I'm not sure what exists that is written in OCaml. I do think there's at least one, I remember someone announcing it on discuss a while back.

[Chris00]

There are a couple of site generators in opam (e.g., stationary, stog, stone) but one would have to check that they are versatile enough for our needs (in particular, executing external tools to evaluate code and fill content — such as news, latest emails, running header, available translations,...). Ideally, before changing the code, presenting a good design for putting all parts of the web site together would be nice.

[agarwal]

I would say the criteria is:

• Use popular static site generators. IMO that rules out all in the OCaml community.
• Be able to call out to scripts (which we of course would want to write in OCaml) and insert the results into the generated code. This is important because as @Chris00 says we have to do some coding for the various custom features we desire.

So the point is the main framework can be in any language. It's just a tool we're calling out to, not implementing or extending. Any code we write however should be OCaml. I agree with @Chris00. We need someone who's done the research to tell us whether such a tool exists and really meets all of ocaml.org's needs.

[Drup]

Last time I got interested, the main problem with the ocaml.org build process was neither omp nor mpp (although they have their issues). The problem was that, and I hope I'm not going to offend anyone, the higher level part is just a huge pile of unprincipled spaghetti. Using a proper static website generator would certainly help, regardless of what is used as template and markup languages/processors. Still, while the perfect tool might exist, I think you are overstating the advantage of switching to it. In all cases, here are my opinion on some of them:

• Omd is a pretty bad markdown preprocessor. It doesn't handle half the features it should, and has not been updated to CommonMark.
• Don't use Jekyll. It's featureful, but the ruby stack is horrible and breaks regularly. I think it's a safer bet to rely on python, for example.
• Stog is, honestly, just a mess of horribly verbose and badly designed xml. It's also extremely over-engineered.
• Stone is rather simple, but pretty decent at handling the high-level part. The detail of each page is handled by "exporters" which don't have to be OCaml programs. The template language might be a bit weak though. Last time I looked, the code was clean.
• I didn't know stationary, but according to the example, it's similar to cowabloga (the thing used for mirage.io): it's a library to generate websites, not a tool. It seems smaller than cowabloga which is already quite small.

There is also @samoht's ramen I guess. I'm not sure how alpha that is. :) It's more or a less a strict extension of stone and very jekyll-like, so that might be worth it.

[samoht]

There is also @samoht's ramen I guess. I'm not sure how alpha that is. :) It's more or a less a strict extension of stone and very jekyll-like, so that might be worth it.

It's very much alpha -- just enough to generate my website atm. The language is a bit had-hoc (as I just added the bits that I needed, trying to keep it general enough) and untyped, which is far from ideal. The main difference with other frameworks is that it allows to compose templates (e.g. use them as functions) which I found missing in other solutions. I would be very interested to have a typing layer and would gladly accept contributions :-) Or I am happy to switch to the new tyxml template language when it's ready!

[Drup]

The tyxml template language will be quite a while in the making, don't wait for it. ramen's template language is sort of halfway between mustache and jinja.

In any cases, the templates are already written in mpp, and it works ok. I'm not sure it's worth changing them. As I said, the problem is the high-level orchestration, not the individual low-level tools.

[bluddy]

Don't use Jekyll. It's featureful, but the ruby stack is horrible and breaks regularly. I think it's a safer bet to rely on python, for example.

Well, another option is to rely on Docusaurus, as mentioned in another issue. Javascript is 'closer' to us due to the Reason connection. If there's nothing mature on the OCaml front, and it's too much effort to make what we have now into something more robust, then that's the way to go. We can create a Docusaurus site (which also happens to have been Reason's choice) and retain only the planet part, perhaps hosting it at planet.ocaml.org. There's probably some way to create a feed aggregator with Docusaurus but I couldn't find it.

One big advantage of Docusaurus is that Facebook will update it to keep up with the latest standards in looks/css/html and such. 'Outsourcing' that work to them makes sense.

[Chris00]

@bluddy Does Docusaurus support calling external tools to generate content? (A simple example in addition to the above: the latest OCaml version is taken from opam and put into a variable to ensure the references stay properly updated site wide.)

[Chris00]

As I said, the problem is the high-level orchestration, not the individual low-level tools.

@Drup I somehow agree with that — even though the tools could benefit from being more robust IMHO — but do you have references on how to do that properly? Indeed, at the moment, the content is in site/ (they can be HTML or Markdown files, with a language extension convention) and the structure of the directories reflect the structure of the site. The scripts to produce content are in script/ (and called through mpp). The templates are in template/. I suppose that by “high-level orchestration” you mean the way these pieces are combined together. At the moment, the relevant part are mostly Makefile.from_html and Makefile.from_md. Is there a “standard” way of doing this — easier to understand while keeping the versatility/automation we need?

[bluddy]

@bluddy Does Docusaurus support calling external tools to generate content? (A simple example in addition to the above: the latest OCaml version is taken from opam and put into a variable to ensure the references stay properly updated site wide.)

I'm not sure. This seems to suggest it may be possible to customize the startup scripts.

I found this issue which would have made it very easy to insert variable values such as versions into all markdown files. Unfortunately it seems like currently it can only be done with javascript files.

[pmetzger]

BTW, one option: it is not necessary that the entire web site be generated with the same tools. It might be that much of the content could be nearly pure markdown with pure static generation, while some stuff like the front page which presents latest news could be more dynamic, or static re-generated on a different schedule.

[agarwal]

The problem was that, and I hope I'm not going to offend anyone, the higher level part is just a huge pile of unprincipled spaghetti.

No offense at all. I agree. It's a mess. My reason for encouraging us to use more popular tools is:

• Avoid all this build mess. I went through the same thing implementing cufp.org, and I was amazed at how much work that part turned out to be. Someone else in some other community must have solved this problem, right? Maybe not, but I'm hoping.

• Force us to follow some common conventions around the content's structure and style. That way we stop spending our own time on this. I just want the whole site to be more simple. We're doing things like handling internationalization, creating breadcrumbs, generating table of contents, etc. All being done by us with code we wrote from scratch. I can't help but feel our energy is being wasted on things others have surely done.

[pw374]

Every few days, weeks, months, or years, tools and technologies become deprecated. To avoid that, we need manpower for maintenance. It does make sense to evolve, pick a new set of tools (a new set that doesn't have to have an empty intersection with the current set), make the whole thing work. But if you (that's a very general "you") are hoping that once it's done, once the right choices are made, once we've re-done the whole thing correctly, we won't have to do it again, then you're probably dreaming even harder than the dreamer of Imagine by John Lennon.

If we use a popular tool, don't hide away from history. Popular tools evolve, or die. When they evolve, we have to put the maintenance efforts to keep things up to date. After they die, we have to pick other tools.

I don't mean to be negative here, I mean to be realistic.

I do agree that the whole ocaml.org system has become too complicated and too fragile. I do agree that there has to be changes, there's a number of issues that are taking way too long to address. Either pick another system, or make the existing system evolve. It could for instance "simply" be simplified, reorganized. But at the same time, let's try to avoid trading our current set of issues for another set that doesn't make things actually better.

My 2 cents.

[pmetzger]

BTW, having played with Jekyll a bunch, I'm wondering if we shouldn't migrate. Yes, that means the site won't be built with OCaml tooling any more, but I'm not sure how important that is.

[agarwal]

At present, I believe the best options are: jekyll, hugo, or docusaurus. But picking the tool isn't really the problem. The lack of manpower is the real issue.

[Chris00]

Also, it is not clear to me whether picking another site generator will help more people to contribute. Indeed, there are more pressing matters (e.g. rewrite tutorials) to spend time upon so it only make sense to do this if it brings more contributors.

[Drup]

@Chris00 Building the website is currently convoluted to the point where I, as an author, can't manage to build it locally to vizualize my changes. Furthermore, the current markup languages is very limited, because it's basically non-compliant markdown without any extensions.

I assure you, those two problems are very big brakes on potential authors.

[Chris00]

@Drup Good to know! ☺

[Drup]

Note that I didn't say the only way to solve these problems was to use off-the-shelve website generators. ;)