Feature: Jekyll-style plugins using Lua

lucperkins commented 5 years ago

IMPORTANT NOTE: I am interested in doing the actual work here. This is thread is more to assess the interest level and get feedback.

SECOND IMPORTANT NOTE: I'm also happy to bring this discussion to Discourse. I'm not sure which venue should have priority for this.

Proposal

One of the things I like about Jekyll is its plugin system, which allows you to weave arbitrary Ruby logic into various points in the build pipeline. I think that the benefits of this for Hugo are pretty clear, and I've seen plugin proposals floated in past issues (#321) and community discussions.

It recently came to my attention in the Tools/libraries that can drive new Hugo features discussion that there's a Lua interpreter written in Go—actually more than one—that could, with some massaging, be used to enable Hugo users to run Lua scripts as part of the Hugo build pipeline.

Basic feature outline

This is obviously majorly up for discussion, but I envision something like this:

Initially plugins would be disabled by default; you'd need to set an --enablePlugins flag to use it
You could can a plugins directory to your Hugo project; at run time, Hugo would scan that directory for Lua files
For an initial implementation, you would name files based on where in the build pipeline you'd want the logic to run. A file called prebuild.lua would run before Hugo does anything, prepage.lua would manipulate each page before Hugo does, postbuild.lua would run after Hugo is done with everything, and so on.
Eventually, you could use these scripts to add new templating functions, create project-specific Markdown extensions, etc. Anything you see here would conceivably be within reach.

Basic example

I'll provide a super simplistic example because I'm not yet super familiar with Lua. Imagine a postpage.lua script that runs after Hugo has processed a page. Imagine if for some reason you wanted to make the title of each page in the blog section lowercase:

-- plugins/postpage.lua
for i, page in ipairs(pages) do
  if page.section == 'blog' then
    page.title = string.lower(page.title)
  end
end

Feedback

Is this something that would be beneficial for Hugo? Does this seem like a vaguely reasonable approach? Worth the effort? Not the right direction for the project? Too much flexibility? Too much power?

bep commented 5 years ago

I think it would helps to tell us a little about the use cases. What common situations today cannot easily be done in Hugo? I' don't think the Jekyll plugin index are very relevant. And your "basic example": I would say that you should do that lower casing in the template for the blog section (or possibly in CSS).

bep commented 5 years ago

@lucperkins I linked this issue into #5455 -- as they are somehow related. Some of the same concerns I had with that also applies to this: Will it scale. I think your "to-lower" example illustrates this: You eventually end up with a set of very fine grained code blocks that all operate on the same data set. Which will create a big, quadractic loop.

I think, however, that this could work, if you turn this into a "subscription based" approach with a well defined concurrency model, e.g.

withPagesMatching("**/blog/**", p) {
   p.title = string.lower(p.title)
}

I stilll think that the above example does not fit in a plugin, but at least with the above model (or something similar), we can probably make it work at scale.

lucperkins commented 5 years ago

@bep That example was admittedly very trite. What it all boils down to is this: I work on lots and lots of Hugo sites, mostly for big technical documentation projects. Despite Hugo's immense—and broadening—power, I always end up writing ancillary scripts to complement Hugo, often in the form of generating JSON or YAML to go in the data directory.

What I would love would be the ability to ditch those ancillary scripts and use a real scripting language to fill in the gaps. One thing I do like about Jekyll is that I know that jekyll build can always do 100% of my bidding, no matter what that may be. Often I can do a Google search and find a plugin that does exactly what I need. curl -O plugin-i-need.rb and I'm all set. I do think that a lot of the enduring appeal of Jekyll stems from its shareable plugins ecosystem. GatsbyJS is also experiencing huge growth right now, I think partially because it's "programmable" all the way down.

To give a better example than the one I gave above, there's been discussion in this repo about enabling Hugo to fetch content from GraphQL-based content APIs like Contentful and GraphCMS. That would be a good use case (and vastly preferred to baking that into Hugo). I can also imagine, for example, fetching a bunch of release information about a software project from the GitHub Releases API and attaching that info to .Site.Data or even just adding a .Release variable that's globally available. Another possibility: adding my own Markdown extensions by manipulating the .Content of a page.

Admittedly, nothing has ever been beyond my reach using what's currently available in Hugo. It's not a "necessary" thing without which Hugo would be incomplete. I just know that the "wow" factor of this could very well be on the same level as Hugo Pipes.

bep commented 5 years ago

I just know that the "wow" factor of this could very well be on the same level as Hugo Pipes.

I agree, but I think this needs a proper "thinking". I'm happy with how Hugo Pipes turned out, but that implementation wasn't obvious. Most issues/discussions about that subject talked about some "file pipeline", which obviously wouldn't not give you the instant refreshes you get today.

I think this discussion wires nicely into a couple of other issues (that I have quietly been building up to with some of the technical upgrades I've been doing (also the current task)):

Adding page metadata from outside of the pages
Adding pages from data sources

It would obviously be golden if I could do both the above in a flexible plugin way that maintains the ... snappines of Hugo.

lucperkins commented 5 years ago

@bep Yeah, the risk of compromising Hugo's snappiness is a very real one. I honestly don't have a clue what the performance profile of Lua-on-Go looks like. A substantial per-page performance "hit" would indeed make Hugo a different, lesser tool. But if the performance is in the neighborhood of Lua modules for nginx, then perhaps we'd be in business.

FYI I'm also looking at this library: https://github.com/yuin/gopher-lua. Seems potentially more well baked than the golua lib. I've at least found it easier to work with.

bep commented 5 years ago

Note that I'm not particular worried about the scripting languages -- they will most likely be plenty fast for their uses, but the plugin API needs to designed with care.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help. If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open. If this is a feature request, and you feel that it is still relevant and valuable, please tell us why. This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

theHacker commented 3 years ago

Just my issue, so I will leave my two cents :)

First a little story for my motivation: I am currently migrating my WordPress blog. I hate Gutenberg and his bugs and I want to be in control of the output, not having 10 different rendings of the same block, because over 2 years there were 10 different buggy implementions.

First I started implementing my own blog software with Kotlin, Spring and used markdown files and a separate YAML files for meta data, such as date, title, categories and tags.

Then I found out about static site generators and researched. I found Hugo and Jekyll as the top runners to have a deeper look into. In the end I chose Hugo, because I am no Ruby guy.

I am now busy for a few weeks, nearly finished having the site looking the same as in WordPress. However I must forfeit WordPress's archives (/yyyy and /yyyy/mm arranged lists by year and month). I searched the web and found these

They are just the same "solutions" I could imagine: Misusing Hugo's taxonomies to archive the archives. Now plugins come to play.

I do not want to repeat data, i.e.

setting each month a fresh taxonomy in the config and
appending year and month in each post's meta data

when the data is already there!

publishDate: 2017-04-22 23:56:00 makes it clear, I want to have this post in the archives sites /archives/2017.html and /archives/2017/04.html. I only need to tell Hugo about that :wink:

For this use-case I would Hugo need to supply me with a means of telling it what URLs to generate. For example

addListener('allUrls') { hugo ->
  // Hugo gives me a list of all URLs it currently would generate
  // I can add/remove to my liking.
  val allMonths = hugo.allUrls
    .map { it.data["publishDate"].toYearMonth() }
    .distinct()

  allMonth.forEach { hugo.allUrls += "/archive/${it.year}/${it.month}" }
  allMonth.map { it.year }.distinct().forEach { hugo.allUrls += "/archive/$it" }
}

(My Lua is a bit rusty, so Kotlin pseudo code)

Maybe I would need to pass Hugo a hint, which template, or this can be done with sections. I am not that deep into Hugo.

Thinking larger you can imagine a lot of different hooks where you can allow the plugin developer to

alter the input (manipulate data, create dynamic data, ...) before Hugo generates
alter the output
even manipulate a HTML tag, before Hugo writes it (use-case for me would be to add href="_blank" to certain outgoing links; my workaround was to switch the markup engine to blackfriday, because the default engine does not has hrefTargetBlank!)
alter the configuration (dynamicly change the config based on environment, thinking maybe of CI/CD)
implement shortcodes as code
...

Hope that gives you a few ideas to continue here, really would like to see a start :) Don't want to threaten, but I already had the idea to switch to Jekyll after I have the site ready, because of this lack of extensibility...

theHacker commented 3 years ago

Just checked my TODO list.

More ideas a plugin can achieve:

Link checker: When generating a link, we can check whether the page exists and print a warning or abort the generation
Ensure code style: A plugin could check, for example whether all div tags with a cell class either have a even or an odd class).
I can write a dynamic generation for title and url. Right now, if I have a specify schema (not just replace space with hyphen), I have to specify the URL in each post (see https://gohugo.io/content-management/urls/#set-url-in-front-matter)
have more control over image processing - right now Hugo generates really ugly URLs :(
- see https://discourse.gohugo.io/t/change-file-name-for-resized-image-resources/14219

gjvnq commented 1 year ago

A few more ideas of stuff we could use plugins for:

Auto generation of permanent identifiers like UUIDs and ULIDs for pages.
Writing redirect files based on permenent identifiers listed on the pages metadata.
Making password-protected pages by encrypting the content and using client side JS to decode it.

gohugoio / hugo