On-demand rendering for Jekyll

[benbalter]

Summary

Right now, to view a single file, Jekyll must render and write to disk, the entire site. For small sites, this is not a problem, as site generation takes milliseconds. For larger and more complex sites, this process requires significant overhead. Instead, for faster previewing, Jekyll could read in all of a site's files, a relatively inexpensive task, and render only the file requested of a given preview. This would allow Jekyll to generate files on demand, leading to faster previewing, or near-instant viewing of specific tags/commits of a repository, meaning Jekyll could easily surface historical documentation for a given software version.

Expected outcome

A Jekyll plugin or merged core pull request that would allow Jekyll to render a single file, including layouts, includes, and plugins, without needing to render the entire site. Bonus points if it integrated with Git and could do the same for a given tag/commit.

Skills

Basic Ruby knowledge. Experience with Jekyll internals (especially the reading and rendering process), or how Jekyll builds static sites, as well as basic dynamic web publishing experience (getting a request, processing it, responding) helpful.

Mentors

@parkr, @benbalter

[cyhsutw]

Hi @benbalter, this project looks interesting. I would like to join this. How can I discuss this with you, via email or just leaving comments here?

[benbalter]

Awesome to hear! Glad to chat here, if you don't mind the conversation being public, otherwise, you can email opensource@github.com and mention this issue, and we can chat privately

[nirlanka]

hi @benbalter, @parkr! This has been something I've always wanted since I started using Jekyll. I've been planning my own simple alternative to Jekyll with on-demand/selective rendering as a key feature. I'd love to start working on this. P.S. I was kind of under the impression lack of on-demand rendering was a key design decision in Jekyll

[benbalter]

@NirmalL That's awesome to hear. While rending the entire site to static files is definitely a design decision for what we might call "production", if there is such a think for static sites, there are two use cases that come to mind, where prioritizing, or selectively rendering may be useful:

1. Previewing locally. If I change about.md, and Jekyll detects that, there's a good chance I'm going to load about.md in my browser to see the changes. In many cases, we can render about.md first, before other posts or pages.
2. Similarly, if I have a git-versioned Jekyll site, say for software documentation, if my project's on version 2.0, but I want to view the docs for version 1.0, once again, as the user requests a particular page, we can render that page on demand, and dependencies, resulting in what appears to be a faster build.

[nirlanka]

@benbalter I see. I think the second scenario can perhaps be quite useful in something like gitpages. Is that the idea? I guess we can use git API to identify the changed files. I guess it might even be extensible to even partial-render the pages... to an extent, though it's probably an overkill. Do you have any suggestions/advice on how we may proceed? I'm planning to go through the things in the Jekyll documentations that seems relevant and related git API.

[benbalter]

I think the second scenario can perhaps be quite useful in something like gitpages. Is that the idea?

That's one idea, yes, but there are lots of use cases, including faster local previewing.

I guess it might even be extensible to even partial-render the pages... to an extent, though it's probably an overkill.

If you think about the process required to view /about for a site, today, we have to generate the entire site. That's fine for a site with 5, 10, even 100 pages, as it'll generate pretty much on demand. As sites get more complex, that delay can become noticeable. But thinking through what's actually happening when you request /about in a browser, you need about/index.html, and maybe a few CSS/JS/image files linked from it, but for the most part, you're only using a tiny fraction of the site on any given request. You can take that further, and then maybe prioritize pages that are linked from that page, and suddenly given an edit to a file, or a request to preview a certain file, we can prioritize the order that we render things, rather than doing so e.g., alphabetically.

Do you have any suggestions/advice on how we may proceed? I'm planning to go through the things in the Jekyll documentations that seems relevant and related git API.

I'd caution against spending too much time digging into the Git API. First, Jekyll itself, although widely used along with Git, is technically Git agnostic. It just reads the file system, and it's easy enough to build prioritized rendering based on the assumption that the file system represents what you want. git checkout is an inexpensive operation. Second, even if we did want to integrate directly with Git, libraries like Rugged largely abstract the API, and GitHub has staff that can help with Git operations, if that's the direction that things go.

[nirlanka]

@benbalter Thanks for the explanations. I think now I understand what you mean. I agree with your points about git too. As a way to proceed would you recommend first analyzing and listing the goals, requirements and sub-features, then prioritize them according to the feasibility and usage?

[ismnoiet]

HI, @benbalter may be i'm a little bit late but i think this is interesting and actually i'm working on some jekyll related things. Can you please give me more details.

[benbalter]

@ismnoiet Not too late at all. You've got until March 25th to apply. Reading through the above thread should be a good place to start, in terms of better understanding the suggested project. Did you have any specific questions or concerns we could help answer?

[ismnoiet]

Thanks @benbalter, yes i have some questions and ideas and i'll be discussing them with you if will very soon.

[MikeMcQuaid]

GitHub is not participating directly in GSoC 2017. The Homebrew project (which I maintain) has applied to participate this summer: https://github.com/Homebrew/Outreachy-and-Google-Summer-of-Code. Note you'll need access to a Mac to take part. Thanks!