Mowyw replacement

[2colours]

So that the work is more visible...

The end goal would be to generate basically the same output files, with the same folder structure.

My main concern is the content and the template processing but I suppose ultimately Plack could also be removed and a plain Raku version could be created.

Another thing is that this could be made more module-ish if need be. Currently it depends on Path::Finder and Template6 implicitly for the template processing, and HTTP::Tiny, JSON::Fast and LibXML::Reader for the blog-grabbing.

[2colours]

I updated the source files/integrated the modifications. I would say it went mostly flawlessly, except the navbar modifications needed to be integrated by hand.

Based on https://github.com/Raku/raku.org/pull/215 I could create a Dockerfile and took over that Caddyfile as well. One can use the very same instructions to get it working locally.

There is one catch: the blog posts. I don't think we want to build a new container every time new blog posts might get added, therefore I rather omitted the whole fetch-blog-posts.raku part, removed LibXML from the dependencies (which was convenient because that requires build-essentials and libxml2-dev to be built into the builder image) and tried to just borrow the API from the current raku.org site. This could work in theory but it won't from localhost because of CORS. Either way, I think the blog posts should be out of the scope of the static site since they aren't very much static. The only reasonable way to add it would be to install a cron job inside the container that keeps updating the list. I would personally prefer a separate container, and overall a separate API provided by planet.raku.org for the blog posts.

[dontlaugh]

It will take some time to study, but in principle I support this change. Two notes:

1. The current raku.org production host is not using containers, and is - in fact - not Linux. That is okay, because providing a container is good for local dev at the very least.
2. Hold off on merging this until @andinus has a look, since the current cron jobs reference mowyw and perl 5, and will break stuff if this gets merged right away.

Current OpenBSD httpd configs

**/etc/httpd.conf** (main config, just serves up static files) ```tcl server "raku.org" { listen on 127.0.0.1 port 280 gzip-static root "/htdocs/raku.org" } server "www.raku.org" { alias "perl6.org" listen on 127.0.0.1 port 280 block return 307 "https://raku.org$REQUEST_URI" } server "raku.org" { alias "www.raku.org" alias "perl6.org" listen on * port 80 gzip-static root "/htdocs/raku.org" include "/etc/httpd.d/acme-redirect-rest.conf" } ``` **/etc/httpd.d/acme-redirect-rest.conf** (partial include). Not relevant for web serving, but just included for completeness' sake ```tcl location "/.well-known/acme-challenge/*" { root "/acme" request strip 2 } location * { block return 301 "https://$HTTP_HOST$REQUEST_URI" } ```

Current cron jobs related to raku.org (updates required!)

``` trinity# su raku.org trinity$ crontab -l # SHELL=/bin/ksh PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:/home/raku.org/.local/bin:/home/raku.org/.raku/bin:/home/raku.org/.perl5/bin:/home/raku.org/.local/share/gem/ruby/3.2/bin:/usr/local/rakudo/bin:/usr/local/rakudo/share/perl6/site/bin:/usr/local/rakudo/perl6/vendor/bin:/usr/local/rakudo/share/perl6/core/bin HOME=/home/raku.org MAILTO=andinus #PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin PERL5LIB=/home/raku.org/.perl5/lib/perl5 PERL_CPANM_OPT=-M https://cpan.metacpan.org/ PERL_MB_OPT=--install_base "/home/raku.org/.perl5" PERL_MM_OPT=INSTALL_BASE=/home/raku.org/.perl5 PERL_LOCAL_LIB_ROOT=/home/raku.org/.perl5 # # minute hour mday month wday [flags] command # 10~15 * * * * -ns cd $HOME/raku.org && perl fetch-recent-blog-posts.pl && cp -rv online/recent-blog-posts.json /var/www/htdocs/raku.org/ 20~25 * * * * -ns cd $HOME/raku.org && git pull && mowyw && perl fetch-recent-blog-posts.pl && cp -rv online/* /var/www/htdocs/raku.org/ 35~40 * * * * -ns cd /var/www/htdocs/raku.org && (yes | find . -type f ! -name '*.png' ! -name '*.gz' -exec gzip -v -k -n -9 "{}" \;) ```

But since the output of this change is still a static site, it will definitely work. Just need to prep prod first.

[andinus]

I was able to get things installed, I followed instructions on https://github.com/2colours/raku.org/blob/mowyw-replacement/README.md#build

When I run publish-sources.raku, I get this error:

raku.org@trinity ~/2colors-raku.org> raku publish-sources.raku
Failed to copy '/home/raku.org/2colors-raku.org/source/bootstrap/css/bootstrap-theme.css' to '/home/raku.org/2colors-raku.org/online/bootstrap/css/bootstrap-theme.css': Failed to copy file: no such file or directory
in sub MAIN at publish-sources.raku line 23
in block <unit> at publish-sources.raku line 4 

Creating the parent directory if it doesn't exist should do the trick. Once we're able to get this script running we should be good to go.

[2colours]

I added safety measures when a path should be guaranteed to be immediately creatable. I'm not at home so I haven't tested it, just went by gut feeling... perhaps I can also test it when I get home.

By the way, @andinus please keep in mind that those instructions, while supposedly more or less up-to-date, are only about getting the website running much like a snapshot, without any particular infrastructure. The traditional hosting was probably solved by the update.sh script, which I also tried to port. (Maybe the repo name would be worth changing though.) That one would check the modifications and update the blog posts and the static content as well.

[2colours]

Fix fixed; of course the parent of the target has to be pre-created, not the parent of the source.

[andinus]

@2colours Looks good now, I am able to build the website. As for updating it, we have a cronjob in place, I'll just have to change the script path (from mowyw to ./publish-sources.raku). It doesn't rely on update.sh.

Ping me when this gets merged, I'll update the cronjob and we should be good to go.

[coke]

Makefile still has mowyw instructions

[coke]

I was able to get things installed, I followed instructions on https://github.com/2colours/raku.org/blob/mowyw-replacement/README.md#build

instructions assume scripts are executable, but they are not (require calling through raku). Also, zef install --deps-only . isn't sufficient to run fetch-recent-blog-posts.raku - fails on LibXML::Reader:ver<0.10.4>:auth<zef:dwarring>:api<0.10.0>

[2colours]

Makefile still has mowyw instructions

There was no Makefile when I created this. :) Anyway, I don't really know why it got added.

instructions assume scripts are executable, but they are not (require calling through raku)

For me they are, and the git history is clean. Andinus also hasn't complained about this.

Also, zef install --deps-only . isn't sufficient to run fetch-recent-blog-posts.raku - fails on LibXML::Reader:ver<0.10.4>:auth<zef:dwarring>:api<0.10.0>

This is known and "deliberate", please read the comment:

There is one catch: the blog posts. I don't think we want to build a new container every time new blog posts might get added, therefore I rather omitted the whole fetch-blog-posts.raku part, removed LibXML from the dependencies (which was convenient because that requires build-essentials and libxml2-dev to be built into the builder image) and tried to just borrow the API from the current raku.org site.

It's hard when the means of deployment aren't clear... also, you can see that the jQuery AJAX call contains the full URL of the site. That can still work if the site is hosted there and fetch-blog-posts.raku is executed regularly but it shows that I basically moved that part out of the scope of the whole change. Ideally, it shouldn't be a part of the site.

[andinus]

For me they are, and the git history is clean. Andinus also hasn't complained about this.

Oh, it actually isn't executable, I must have ran raku publish-sources.raku.

I ran it on in place of existing site, that explains why I didn't notice LibXML not being in dependencies.

There is one catch: the blog posts. I don't think we want to build a new container every time new blog posts might get added [...] It's hard when the means of deployment aren't clear...

Makes sense, we can have it as a separate script. Currently we're running fetch-blog-posts.raku independently at set intervals. And the website is also being compiled on the box itself. It is not using containers.

[2colours]

I don't know why file permissions aren't carried through; I have definitely seen git be able to deal with it... but anyway, as a safety measure, I will update all occurrances of "just invoke this executable" to invoking the runtime directly.

With the deployment-related matters, I will need some help:

• do we want a container or not? (Dockerfile, Caddyfile)
• what about the Makefile? Do we need it, and what for?
• when is the server updated? Currently, it states "every 15 minutes" but I have no clue if that was ever correct
• Is there anything that still uses Perl? (plackup and friends) How is the site actually set up? Do we need files like app.psgi and update.sh at all?
• eventually, the blog post retrieval needs to be leveraged some way; right now I'm okay with the fetch-blog-posts script and some cronjob that updates it but it's quite suboptimal

[andinus]

Márton Polgár @ 2023-11-26 06:45 -08:

With the deployment-related matters, I will need some help:

• do we want a container or not? (Dockerfile, Caddyfile)

@coleman suggests using it for development workflow. It's not used for deployment.

• what about the Makefile? Do we need it, and what for?

It's might convenient to type make rather than 3 separate commands. Not necessary though.

• when is the server updated? Currently, it states "every 15 minutes" but I have no clue if that was ever correct

Currently it's updated every hour at random minutes. We can change that to every 15 minutes.

• Is there anything that still uses Perl? (plackup and friends) How is the site actually set up?

This is how it's setup currently:

cd $HOME/raku.org && git pull && mowyw && perl fetch-recent-blog-posts.pl && cp -rv online/* /var/www/htdocs/raku.org/

After merge I'll change that to:

cd $HOME/raku.org && git pull && raku publish-sources.raku && raku fetch-recent-blog-posts.raku && cp -rv online/* /var/www/htdocs/raku.org/

Edit: We need update.sh. We might not need app.psgi.

Unless we need to serve with Plack, in that case we can setup a reverse proxy. Currently Plack is also just service online/ directory (referring to app.psgi).

[2colours]

@dontlaugh ping 🥺