Investigate making content files processing more efficient

[stelcodes]

I need to test this further but I believe the content files might be getting processed more times than necessary. render-content may be called more than once by end-users for starters. That's a rare use-case but still worth looking into. render-content could just return a precomputed value via closure. This might be the best path forward.

Alternatively I could start sticking Hiccup in the Nuzzle config. I could add a :nuzzle/rendered-content or :nuzzle/content-hiccup key. That way users could look at the content hiccup via tools like Portal easily with nuzzle.api/transform. At that point I might as well stick the whole rendered hiccup in the page entries too under a key like :nuzzle/page-hiccup! That would mean that the user's render-page function is now part of the config transformation process. Not sure!

[stelcodes]

Just remembered: users can already look at the Hiccup content of any page easily by using nuzzle.api.transform:

(def x (-> transform [:blog :foo] :nuzzle/render-content))
(x)

[stelcodes]

Currently the only inefficiency apart from the possibility of users calling render-content more than once is that the Atom feed generation will call it again to fill in the entry content tag:

user=> (publish)
21:00:36 INFO ❌🐈 Removing drafts
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x62aa6a04 markdown/babashka-postgresql-backups.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x38f6715e markdown/functional-news.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0xd58367d markdown/grid-layout-css-eric-meyer.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x70ffa525 markdown/getting-clojure-russ-olsen.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x4b60df52 markdown/self-care-android-app.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x1047b592 markdown/i3-or-sway-nixos.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x654a9bc0 markdown/developer-blog.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x21a2e341 markdown/using-directus-cms.md]
21:00:36 INFO 💫🐈 Publishing static site to: /home/stel/code/dev-blog/out
21:00:36 INFO 💎🐈 Using overlay directory: /home/stel/code/dev-blog/overlay
21:00:36 INFO 📰🐈 Creating Atom feed file: /home/stel/code/dev-blog/out/feed.xml
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x38f6715e markdown/functional-news.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x4b60df52 markdown/self-care-android-app.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x21a2e341 markdown/using-directus-cms.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x70ffa525 markdown/getting-clojure-russ-olsen.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0xd58367d markdown/grid-layout-css-eric-meyer.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x62aa6a04 markdown/babashka-postgresql-backups.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x654a9bc0 markdown/developer-blog.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x1047b592 markdown/i3-or-sway-nixos.md]
21:00:36 INFO 📖🐈 Creating sitemap file: /home/stel/code/dev-blog/out/sitemap.xml
21:00:36 INFO ✅🐈 Publishing successful

[stelcodes]

With PR #117 now the content files are only processed once when publishing no matter how many times render-content is called!

user=> (publish)
21:19:40 INFO ❌🐈 Removing drafts
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x6261f016 markdown/using-directus-cms.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x5b0f00bd markdown/developer-blog.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x7c93ac14 markdown/i3-or-sway-nixos.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x4196c764 markdown/self-care-android-app.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x210161fc markdown/getting-clojure-russ-olsen.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x2b7b8055 markdown/grid-layout-css-eric-meyer.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x10d8379a markdown/functional-news.md]
PROCESSING MARKDOWN FILE:  #object[java.io.File 0x3aa67f0a markdown/babashka-postgresql-backups.md]
21:19:40 INFO 💫🐈 Publishing static site to: /home/stel/code/dev-blog/out
21:19:40 INFO 💎🐈 Using overlay directory: /home/stel/code/dev-blog/overlay
21:19:40 INFO 📰🐈 Creating Atom feed file: /home/stel/code/dev-blog/out/feed.xml
21:19:40 INFO 📖🐈 Creating sitemap file: /home/stel/code/dev-blog/out/sitemap.xml
21:19:40 INFO ✅🐈 Publishing successful
nil

[stelcodes]

There is a core inefficiency in the create-site-index function. In order to know if a page entry should be included in the website, the page has to be at least rendered to Hiccup by the user's render-page function in order to see if the resulting value is nil. Stasis throws errors with a value of nil for a page. If Stasis just pretended like the site-index entry wasn't present if the value was nil, then we could defer calling the render-page function until we needed it. This is really only about making the development server faster. Of course the site publishing will call the render-page function on all the page entries regardless.

The easiest way to change Stasis would be by copy-pasta into this repo. I could pretty easily condense the source into a single small file and maybe put it a new namespace called nuzzle.stasis.

A consequence of this change would be that #117 should probably be reverted. Instead of having the render-content return a static value, it should become lazy again. The publishing pipeline could cache the results in an added page entry key to avoid reprocessing. The user could still call their render-content functions twice but this would be highly unlikely.

Right now the server has to turn every Markdown file into Hiccup and call the user's render-page function for every page on every request. This is costly and means that the server will get slower and slower as the website grows.

[stelcodes]

Ideally the render-page-index function could look like this:

(defn create-site-index
  "Creates a map where the keys are relative URLs and the values are a string
  of HTML or a function that produces a string of HTML. This datastructure is
  defined by stasis."
  [{:nuzzle/keys [render-page] :as config} & {:keys [lazy?]}]
  {:pre [(fn? render-page)] :post [(map? %)]}
  (reduce-kv
   (fn [acc ckey cval]
     (if (vector? ckey)
       (assoc acc (:nuzzle/url cval)
              ;; Build pages on the fly by lazily calling render-page
              (if lazy?
                (fn [_]
                  (log/log-rendering-page cval)
                  (some-> (render-page cval) hiccup/html-document))
                (some-> (render-page cval) hiccup/html-document)))
       ;; If not a page entry, skip it
       acc))
   {} config))

[stelcodes]

I opened a feature request issue in the Stasis repo.

[stelcodes]

With the new :nuzzle/ignore-pages config option, the markdown processing efficiency is MUCH better. Now the render-page function is only called once per request when using Nuzzle's development server. I realized this way would be a better way to control which pages shouldn't be rendered than returning nil from the render-page function.

[stelcodes]

Just confirmed that Markdown processing is now very efficient! For publishing, each content file is processed once. For the development server, pages are processed lazily so render-page is called only once and the content files for that call are the only ones processed. Awesome!