Virtual Templates: more control over collisions between physical and virtual templates?

[danburzo]

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

With fixing #1612, Eleventy gains the ability to produce ad-hoc, Virtual Templates that work as if they existed as physical files on the disk. Right now (if I understand correctly), Eleventy throws when a virtual template collides with an existing physical template.

This can be guarded against with file-system checks (e.g. fs.exists()), but if Eleventy has a good idea that a physical file already exists, maybe there’s an opportunity for a more graceful API. I’m starting this issue to explore possible directions for this feature.

Describe alternatives you've considered

No response

Additional context

No response

[zachleat]

“‘first one wins, last one wins, die on error’ with the latter as a default would be super dope”— @eaton

https://fosstodon.org/@eaton@phire.place/112281097866265239

[eaton]

FYI, the use case that I've been wrestling with for a while in 11ty is slightly tangly, but I believe it's relevant to the issue. I wrote a plugin that

1. Pulls items from a SaaS content API
2. Makes each item available as part of the global data, keyed by the unique content ID
3. Builds different utility collections corresponding for each content type ('articles', 'author bios', etc)

For "special" pages like the home page, customized landing pages, etc, a uuid front matter property combined gets detected and populated with the full content object. For bulk pages (articles, etc) a single paginated template just loops over one of the plugin-built collections.

This works great, EXCEPT when special high profile articles need special presentation/layout treatment. In a perfect world, I'd love to simply drop a custom template into the 11ty directory, add the uuid property to its frontmatter, and have it "take over" that piece of remote content. But because there's no easy way to perform complex build-time alteration of the pagination collection (ie, "process every item that doesn't collide with an existing on-disk output path"), the output paths of the custom template and the paginated record collide and everything grinds to a halt.

I need to do some testing with alpha-6, but using the new "inject a custom template" feature seems like it would simplify some of the work: programmatically inserting a custom template for each content item would allow the plugin to ensure no on-disk templates have "claimed" the content ID before proceeding.

Alternately, allowing finer grain control over the template and output path collision handling could also make these case easier — allowing a paginated page to supply "fallback" versions of pages the can be over-written, for example.

[danburzo]

In a perfect world, I'd love to simply drop a custom template into the 11ty directory, add the uuid property to its frontmatter, and have it "take over" that piece of remote content.

This would manifest as something like below, right?

src/posts/my-post.md

---
uuid: 0000-0000-0000-0000
layout: 'layouts/my-custom-layout.njk'
---

// .eleventy.js
eleventyConfig.addTemplate('posts/my-post.md',  "my remote content", {
  uuid: '0000-0000-0000-0000',
  some: 'data',
  more: 'data'
})

In essence, I think of a Virtual Template as providing two pieces of data to an input path:

• the raw template content (rawInput)
• front-matter data that participates in the Data Cascade

The collision on an addTemplate() call can have one of these outcomes (more or less corresponding to Node’s system flags):

• write always writes the template, overwrites existing.
• add writes template, but throws on existing.
• skip writes template if it doesn’t already exist.
• merge writes template, merges with existing template.

So, one declarative approach would be to state your desired outcome, with a method signature like:

addTemplate(path: String, content?: Object, metadata?: String, mode?: String = 'add')

The data types for each argument neatly allow us to have these shortcuts:

// Metadata without content
addTemplate(inputPath, { 
    some: 'data',
    more: 'data'
}, 'merge');

// Content without metadata
addTemplate(inputPath, 'Some content', 'merge')

An alternate, imperative approach:

• a hasTemplate() method to check, and act upon, a possible conflict
• the addTemplate() method would always work in write mode, overwriting existing info
• an additional mergeTemplate() method that works in merge mode to merge in either content or metadata or both.

This set of methods would allow us to express any collision outcome:

if (config.hasTemplate(inputPath)) {
    // mode = write
    config.addTemplate(inputPath, ...);
    // mode = add
    throw new Error('template already exists');
    // mode = skip
    continue;
    // mode = merge
    config.mergeTemplate(inputPath, ...);
}

[danburzo]

A third way to handle collisions, and possibly the most flexible without any additional API surface area would be the callback variant:

config.addTemplate(inputPath, function(content, data) {
   return ({ content: ..., data: ... });
});

• deciding what to do with existing content and data covers the write and merge modes;
• throwing inside the function covers the add mode;
• returning null or undefined equates to skip mode.

[theTaikun]

With the pre-release of 3.0.0-alpha.15, I too ran into this. I'm attempting to use virtual templates to make themes, where the theme provides a default layouts, but a user should be able to override them by providing their own version of files.

However, the following results in the virtual template being used instead, and the local file being ignored.

.eleventy.js

module.exports = (eleventyConfig) => {
    eleventyConfig.addTemplate(
        './_layouts/default.html',
        "<h1>This is the virtual template layout</h1>{{ content }}"
        )
    return {
        dir: {
            input: "src",
            layouts: "_layouts"
        }
    }
}

src/_layouts/default.html

<h1>This is the local layout</h1>
{{ content }}

src/index.md

---
layout: default
---
This is some content

[eaton]

After some extensive digging around and walking through the way virtual templates are handled in alpha-18, I've got some thoughts.

1. Currently, there's no way to ensure that the path of a virtual template doesn't collide with the path of an on-disk template.
2. When a plugin is preparing to add a virtual template, it can check whether another plugin has already added a virtual template with a given path. It can't override that template, but it can use an alternative path to avoid the collision.
3. At the time these checks are being calculated, the list of on-disk templates hasn't yet been built, and collections.all is empty. It might be possible to change that, but I suspect it would require some extremely deep voodoo in the eleventy build process.
4. Given the fact that the list of on-disk templates hasn't been built yet, there's definitely no way to check other aspects of the templates that are on disk. (For example, preemptively checking for permalink collisions or avoiding adding a virtual template if any on-disk template has a particular front matter property set.)
5. In theory, a plugin could iterate through all the templates on disk on its own, and even load/parse their front matter, to do (3) and (4) on its own. That could have some significant performance implications, though.

TL;DR:

1. Adding some sort of 'ignore/overwrite/error' mode to addVirtualTemplate would make it possible for plugins to force their virtual templates into the mix even if the file path is already taken. However, the ability to check whether a virtual template already exists at a given path means that well-behaved plugins can already back out if a path is claimed.
2. A mechanism to avoid collisions with on-disk templates would be great, but since the list of on-disk templates is built later it would have to be deferred until later in the build process. IE, there would be no good way for a plugin to detect a template collision before adding one without checking the disk explicitly.
3. Ultimately, it might be more useful to add a collision-handling mechanism for permalinks: using custom paths scoped with the plugin name and overriding the permalink can sidestep collisions in a pinch, but permalink collisions are unavoidable. It's almost certainly outside the scope of THIS issue, however.

[paulrobertlloyd]

I think I may have stumbled across the opposite problem, not with collisions, but misses.

I’ve added virtual templates to my plugin, and got them working so that they respect the includes/layouts directory specified by the user in their Eleventy config (PR: https://github.com/x-govuk/govuk-eleventy-plugin/pull/343)

These paths are relative to the input directory; should a path be specified that is outside the input directory, I get errors saying that the specified layout cannot be found.

So this works (rough approximation of the relevant bits of config):

eleventyConfig.addTemplate('layouts/foo.njk', `{{content}}`)

dir: {
  input: "docs",
  layouts: "layouts"
}

But this causes errors:

eleventyConfig.addTemplate('../layouts/foo.njk', `{{content}}`)

dir: {
  input: "docs",
  layouts: "../layouts"
}

// => You’re trying to use a layout that does not exist: layouts/foo (via `layout: foo`)

Is this a bug, or expected behaviour?