gep13 commented 7 years ago

Given a set of photos, it would be good to be able to generate a gallery for showing the photos in say a blog post.

As discussed on Gitter.

gep13 commented 7 years ago

@daveaglick I have been giving this some thought...

I found this:

https://flickrembed.com

Which results in quite a nice gallery in the blog post:

http://www.gep13.co.uk/blog/calgary-zoo

Am I right in saying that I could create a Wyam Module (terminology might be wrong here) that would look at a blog post for a special string, perhaps something like %flickr:72157673850557294% where the number if the flickr photoset number, and then have it be replaced with the necessary html/script to include the gallery in the blog post?

Or, did you have something else in mind when I suggested a gallery the other day?

daveaglick commented 7 years ago

That was along the lines of what I was thinking - there's a number of great JavaScript-based gallery widgets that could be implemented in this way.

A broader challenge is coming up with a systematic way for modules to deal with JavaScript. I've so far been hesitant to add modules like this, not because I don't think they're valuable (they certainly are), but because I haven't come up with a good mechanism. Specifically, there's usually up to three parts of a JavaScript component that has to end up in the final HTML:

The <script> tag in the <head> that loads the JavaScript library.
Any markup or script content for the component (many components these days seem to take the approach of looking for a specific CSS class on a container like a <div>).
Any required JavaScript initialization logic, often placed at the bottom of the page.

Coordinating the insertion of all three parts from a module with either a theme or arbitrary templates turns into a question of knowing where to stick the content inside each input document.

You could also make the argument that the rendering of something like a gallery is entirely a theme/template concern and the module should just be responsible for providing the necessary data. This approach has a number of advantages like allowing different themes to use totally different JavaScript gallery libraries. In this mode, the module would just create a series of documents with information about the images that should go in the gallery. Then a theme would be responsible for inserting whatever JavaScript was needed to show a gallery from that information.

Now that the creative juices are flowing though, I'm also thinking about a totally new way of handling this. Something like a single Snippets module that would load snippets and perform substitution of content for matches. You could define snippets in various ways (in the config file, using a library, maybe even using YAML, JSON, or text files) and there would be a bunch built in. The Snippets module would take the input documents, look for any snippet keys in their content, and replace those keys with the snippet text combined with any arguments. The snippet could look something like what you've put above %flickr:72157673850557294%. We'd need support for multiple arguments to the snippet, escaping the argument delimiter, etc.

A nice benefit of this is that it would be totally template engine agnostic. I plan on integrating Liquid, Handlebars, and others eventually and I'd want this to work across them all. With the right syntax, it would also be independent of HTML - I.e., I wouldn't want to make something like TagHelpers that relies on an assumption about the syntax of the content to work like that it's HTML.

This idea could be really cool - let me think on it a bit more...

daveaglick commented 7 years ago

I'm starting to form a design for this, and the more I think about it, the more I like the idea of snippets. It solves a number use cases where I don't think modules are quite appropriate. In addition to galleries, the concept could be applied to commenting systems like Disqus, Google Analytics, or really any JavaScript library.

Here's my current thinking (want to get it down before it all leaks out of my head):

Snippets will be instances of the ISnippet interface.
This interface will have a single method string Render(string[] args, IDocument doc, IExecutionContext ctx).
Implementing classes will be discovered similar to IModule implementations, including being able to put them directly in the config file or in NuGet packages.
The name of a snippet will be the implementing class name (just like modules).
The config file will also provide a way to specify "simple" snippets:
- Snippets.Add(string name, string result)
- Snippets.Add(string name, ContextConfig result)
- Snippets.Add(string name, DocumentConfig result)
For class-based snippets, the snippet class will be instantiated once per document and it can maintain state over multiple invocations of the snippet within a given document. This will let snippets do things like increment counters or manage stacks over a single document.
Snippets replacements will occur when a new Snippets module is run. The new module will be responsible for finding snippets in the content of each input document and replacing them with the rendered snippet result.
- The module will probably do this by creating a new forward-only stream reader that will identify snippets, buffer them until the closing token is found, and then perform the replacement. That will let snippets apply to documents that just provide a Stream without having to copy the whole stream to a string.
- The new module will be added to each recipe where appropriate.
Snippets will be used with syntax similar to `%%MySnippet -opt 1 -foo "bar baz" something%%
- Snippet name matching should be case-insensitive
- This is intentionally similar to CLI-style argument passing to match the CLI and pre-processor directives
- We'll need to provide escape conditions for both the opening %% and if %% needs to appear inside the snippet arguments.
Wyam.Core/Wyam.Common will contain support for snippets and the Snippets module. A library of actual snippets will go inside a Wyam.Snippets library (or maybe they'll go in Wyam.Html since all the initial snippets will be focused on JavaScript and HTML) that contains a dependency to the command line parsing library. This library will be included in all appropriate recipe packages.
Several of the initial snippets may render to different parts of the page.
- For example, %%gallery head%% might render the <script ...> tag that goes in the head of the page and %%gallery%% might render the actual gallery logic.

daveaglick commented 7 years ago

I changed the issue title to track the broader implementation, but we'll make galleries one of the initial snippets.

gep13 commented 7 years ago

@daveaglick all this sounds good to me! 👍

LokiMidgard commented 7 years ago

will snippites also support something content dependent like:

%%BEGIN%%
      Some content
%%END%%

So the snippet will have access on whatever is between those two tags (in this case Some content), simular to a div tag in html.

Currently I use html tags in markdown and the client side JavaScript is replacing this tags with a more complex structure that allows hiding and animating the content. But compile time support would be much better.

daveaglick commented 7 years ago

Note to self - take a look at the Hexo tag plugins which are quite similar to this concept: https://hexo.io/docs/tag-plugins.html

Specifically, there's already a request for something like the gist plugin

daveaglick commented 7 years ago

Another note to self: snippets need a way to add files to the output folder. Use case is a snippet for a JavaScript widget that requires a separate JavaScript file. Two ways to handle - the Snippet module could either write to the output file system, or it could output additional documents intended to be passed to WriteFiles (currently prefer the former, but will need to think about it).

daveaglick commented 7 years ago

As discussed on Gitter - syntax should be really simple to make easy snippet cases short and sweet. For example, should favor something like %%gist:id%% over %%gist -GistIdentifer XXX%%. One thought is to make the first : a delimiter between the snippet name and just pass the rest to the snippet implementation to handle however it wants to (or omit the colon to pass an empty or null string to the snippet).

daveaglick commented 7 years ago

Another interesting idea from Gitter - recipes/themes could support a global metadata array value called something like JsSnippets. Then you wouldn't even need to add the snippet to the content - the theme would do that by iterating the global metadata collection and adding the snippets for you in the appropriate place.

daveaglick commented 5 years ago

It's well past time to review this issue again. I've renamed it to shortcodes since that term is now used by both Wordpress and Hugo and seems to be gaining general acceptance. Not to mention snippets already has another meaning when referring to code (see #692 and #173).

Here's my current thinking on design taking into account everything discussed above:

Shortcodes will be instances of the IShortcode or IContentShortcode interface.
IShortcode will have a single method string Render(string[] args, IDocument doc, IExecutionContext ctx).
- IContentShortcode will have a single method string Render(string[] args, string content, IDocument doc, IExecutionContext ctx).
Implementing classes will be discovered similar to IModule implementations, including being able to put them directly in the config file or in NuGet packages.
The name of a shortcode will be the implementing class name (just like modules).
The config file (until we're not using a config file anymore) and engine will also provide a way to specify ad-hoc shortcodes:
- Shortcodes.Add(string name, string output)
- Shortcodes.Add(string name, ContextConfig render)
- Shortcodes.Add(string name, DocumentConfig render)
- Shortcodes.Add(string name, Func<string[], object> render)
- Shortcodes.Add(string name, Func<string[], IExecutionContext, object> render)
- Shortcodes.Add(string name, Func<string[], IDocument, IExecutionContext, object> render)
- Shortcodes.Add(string name, Func<string[], string, object> render)
- Shortcodes.Add(string name, Func<string[], string, IExecutionContext, object> render)
- Shortcodes.Add(string name, Func<string[], string, IDocument, IExecutionContext, object> render)
The shortcode class will be instantiated once per document and it can maintain state over multiple invocations of the shortcode within a given document. This will let shortcodes do things like increment counters or manage stacks over a single document.
- Delegate-based shortcodes won't maintain any state.
Shortcode replacements will occur when a new Shortcodes module is run. The new module will be responsible for finding shortcodes in the content of each input document and replacing them with the rendered shortcode result.
- There's an open question of when shortcode replacement should be performed.
- If it happens before templates are rendered then the output can contain markup for the template (I.e., Markdown content).
- If it happens after templates are rendered then it has to be "final" output and there's a question of how to escape shortcodes if they conflict with the template, both the definition with parameters and the content.
  - Maybe template engines should be shortcode-aware...how to enforce generally though?
  - One way this could work is if all template engine modules scan for shortcodes, replace them with an appropriate placeholder that won't get processed but will make it through to the template output, run the template, and then swap back in the shortcode. Hard to envision that working well though.
- Maybe the shortcode itself or the usage pattern decides - I.e., Hugo uses one syntax when the content should be processed by a template and a different one when it should be raw.
- Consider integration into the phased pipelines concept in #695 - perhaps there should be an implicit Shortcodes phase in between Process and Render (which could be opted-out). Or maybe a more general Prepare or Prerender phase where the Shortcode module would generally go.
- The module could perform replacement by creating a new forward-only stream reader that will identify shortcodes, buffer them until the closing token is found, and then perform the replacement. That will let shortcodes apply to documents that just provide a Stream without having to copy the whole stream to a string.
- The new module will be added to each recipe (and Splashdown) where appropriate.
Shortcodes will be used with syntax similar to %% MyShortcode foo bar %%
- Shortcode name matching should be case-insensitive (so similar names will need to be deconflicted).
- We'll need to provide escape conditions for both the opening %% and if %% needs to appear inside the shortcode arguments.
- IContentShortcode and related delegates require opening and closing (designated with /) with content in between:
- %% myshortcode foo bar %% fizz buzz %%/ myshortcode %%
- %% myshortcode foo bar /%%
- I'm not in love with the %% syntax and would welcome alternatives with rationale:
- Hexo uses {% .. %}
- Hugo uses {{% ... %}} to indicate a shortcode that should be processed by the template engine and {{< ... >}} to indicate one that does not need further processing.
- Wordpress uses [ ... ]
- Keep in mind this may need to play nice with various rendering engines depending on when the module gets run
Shortcode parameters are space delimited and parameters with internal spaces can be enclosed in quotes.
- Parameters can be either positional or named, but need to consider syntax for named parameters:
  - Could be %% foo -fizz buzz %% (like CLI parsing).
  - Could be %% foo fizz=buzz %% (like XML attributes).
  - Could be totally up to shortcode implementation (probably want a standard though).
- Consider providing a Shortcode and ContentShortcode abstract base class that includes some standard parameter parsing support.
  - Consider making argument parsing in the Shortcode class static, put it in a helper instead, or add it to IExecutionContext so that the delegate style shortcodes can use it too.
Wyam.Core/Wyam.Common will contain support for shortcodes and the Shortcode module.
- A library of shortcode implementations will go inside a Wyam.Shortcodes library (or maybe in Wyam.Code, or both, depending on how many external dependencies we end up using).
Several of the initial shortcodes may render to different parts of the page.
- For example, %% gallery head %% might render the <script ...> tag that goes in the head of the page and %% gallery %% might render the actual gallery logic.
Shortcodes may need a way to add files to the output folder. The use case is a snippet for a JavaScript widget that requires a separate JavaScript file. Undecided if this will be a part of initial release.
- The shortcode module could write to the output file system directly.
- The shortcode module could output additional documents.

Open questions

What should syntax be - is %% good?
When should Shortcode module be run - before or after templates (or both/either)?
Should shortcodes be allowed to output their own documents or write to file system?

daveaglick commented 5 years ago

I continue to think hard about syntax, particularly as it relates to when the Shortcode module is run and if the shortcode should “pass through” any template rendering (or not). This is a challenge because if you don’t want a template renderer to interfere with the shortcode definition or it’s content so it hits the shortcode module verbatim afterwards, you need to make the shortcode invisible to the template renderer. Being able to do so is necessary for shortcodes that need access to the post-rendered content of the page, for example a “table of contents” shortcode that looks for all headers on the page and constructs a table of contents. If that shortcode is evaluated before the Markdown engine, it won’t know anything about the final HTML content and headers.

Here’s my current thinking (I’ll update the design comment above too once I’m happy with this part of the design):

Shortcodes use a syntax similar to Hugo for consistency, but note that the behavior is quite different. The start and end delimiters are{{ and }} by default but can be customized using the ShortcodeStartDelimiter and ShortcodeEndDelimiter settings (for the remainder of the description below, the default delimiters will be used for illustration).

Content

Some shortcodes support content. This is indicated by pairing shortcode declarations with the same shortcode name and using a slash in the closing declaration:

{{% myshortcode %}} some content {{%/ myshortcode %}}

Any shortcode that either doesn’t support content or doesn’t require it for a specific usage should be closed with a slash on the closing delimiter:

{{% myshortcode /%}}

If you include content in a shortcode that doesn’t support it, no error will be generated and the content will be ignored.

Parameters

Some shortcodes support parameters. These follow the shortcode name inside the shortcode definition and can follow whatever convention the shortcode specifies.

{{% myshortcode param1 param2 /%}}

If you include parameters in a shortcode that doesn’t support them, no error will be generated and the parameters will be ignored. However, if the shortcode is expecting parameters and you specify them incorrectly, an error will be generated.

Evaluation Phases

Shortcodes can be evaluated before template engines (“prerender”) or after template engines (“postrender”). There are three different shortcode syntax conventions depending on the phase in which you want the shortcode to be evaluated and how it interacts with the content around it.

Make sure to pay attention to the intended phase for a given shortcode. Specifying a shortcode in an unintended phase won’t directly generate an error but will probably result in unexpected behavior and results. For example, if a shortcode outputs Markdown it’s probably intended to be evaluated in the prerender phase. Evaluating it in the postrender phase won’t hurt anything, but it also won’t evaluate the Markdown as such because the Markdown engine will have already been run.

Prerender Syntax

Sometimes shortcodes need to be run before any other template processing. These are specified using a % character inside the delimiters: {{% ... /%}}. For example, if you evaluate a shortcode during this phase and it results in Markdown output, that Markdown output will be further processed by the Markdown engine. Because these shortcodes are evaluated very early in the process before any other template engines are run, you generally don’t need to worry about how the shortcode syntax will interact with any template engines that might process the document content.

Postrender Syntax

To indicate that a shortcode should be evaluated after template engines have been run, you use ! inside the delimiters: {{! ... /!}}.

Postrender Commented Syntax

One of the challenges of evaluating shortcodes after template rendering is that the template engine might try to interpret the shortcode definition and any content as part of the template. One way to avoid this is to surround the shortcode in a comment appropriate to the template engine(s). This can be indicate with a * character inside the shortcode delimiters: {{* ... /*}}. This type of shortcode will remove any text immediately before and after the shortcode up to the nearest white space when the shortcode is evaluated.

For example, the following shortcode will be ignored by the Razor template engine even though it contains Razor syntax because it’s surrounded by Razor comments:

<div> @*{{* myshortcode *}} @foo {{*/ myshortcode *}}*@ </div>

Assuming the shortcode just outputs whatever content it contains, this would result in the following:

<div> @foo </div>

By default this convention will not remove the proceeding and trailing whitespace. If you want the shortcode evaluation to also remove the surrounding whitespace characters, you can use #:

<div> @*{{# myshortcode #}} @foo {{#/ myshortcode #}}*@ </div>

This will result in the following:

<div>@foo</div>

Escaping Shortcodes

In some rare cases you may need to escape the shortcode entirely because you don’t want to evaluate the shortcode as such. In this situation you can use the special Raw shortcode which outputs the shortcode content verbatim.

For example:

{{% raw %}}{{% foobar /%}}{{%/ raw %}}

will result in output of:

{{% foobar /%}}

Nesting Shortcodes

Shortcodes cannot be nested. Any shortcodes inside the content of other shortcodes will be evaluated as text content of the outer shortcode. Note that if the outer shortcode is evaluated in the prerender phase and results in output that contains a postrender shortcode, the postrender shortcode in the output will be evaluated as such in the postrender phase.

_{Sent with GitHawk}

daveaglick commented 5 years ago

Another update: after working on implementation, I've confident I could do what's described above. However, I'm much less certain I should do it. I'm now thinking it's overly complex and will create more confusion than a less-robust solution.

I'm now heavily leaning towards only running shortcodes before template rendering (including Markdown). No special syntax for pre vs post rendering. This will limit some of our use cases - for example, we won't be able to write a shortcode like {{% toc %}} that analyzes the headers in a document because it'll get evaluated before there's even any HTML when all the headers are Markdown like # or Razor has a chance to include partials, etc. The tradeoff is a simpler feature that's easier to understand, use, and implement that does one thing (text macros) and does it well.

Some other points:

Shortcodes won't be able to examine content of the entire page, only what's inside the shortcode (this allows them to be processed entirely in a single pass through the content stream).
Shortcodes do get the IMetadata (but not the entire IDocument) being processed and can specify additional metadata to be added to the document.
- This still lets us do a few clever things like make a shortcode add to a Scripts metadata value that eventually gets used by Razor to output scripts in the <head> for example (in which case the Scripts metadata value would be a convention supported by the recipe/theme)
- Any additional metadata becomes immediately available to following shortcodes, but doesn't result in a new IDocument until all shortcodes are processed

daveaglick commented 5 years ago

Some more quick thoughts as I work on it:

Now thinking the shortcodes module should be run after all template processing - that’ll give it the most flexibility to interact with templates. For example, if the shortcode content contains Markdown, the shortcode start and end tags should “fall through” but the content will get processed as Markdown.
Strongly considering XML/HTML processing instruction syntax since Markdown is more lenient with what’s in the tag and it should fall through all templates. Something like <?% name foo bar ?>xyz<?%/ name ?>
A special shortcode named = that outputs whatever is in a metadata value would be cool: <?%= key /?>. If there’s shortcode content, it could be the fallback if the key isn’t found. If the metadata value is enumerable, each item should be output (how to specify delimiter?)

LokiMidgard commented 5 years ago

I'm not sure if I get this right. Shortcode will not be an module that runs somewhere in a pipeline.

Otherwise the execution time would be defined by where in the pipeline it is added. I think my shortcoming has its origin that I had never worked with templates, which are mentioned in your descriptions.

So how could I integrate it in my wyam build, if I don't use templates?

daveaglick commented 5 years ago

@LokiMidgard

I'm not sure if I get this right

As I continue to work on this feature, the thoughts in the issue here keep falling out of date.

Shortcode will not be an module that runs somewhere in a pipeline.

That's not true anymore - shortcodes will absolutely be a module (called ProcessShortcodes). It'll be run after Markdown, Razor, etc. template processing in the recipes, though you could run it at any point in your pipeline like any other module when writing a custom config. There are tradeoffs to running it before vs. after template processing.

So how could I integrate it in my wyam build

The module scans the content of all input documents looking for shortcodes with the syntax <?# shortcode_name other arguments ?>. If any are found, it'll execute the shortcode(s) and replace with definition with the shortcode result content. In a custom config you could run this against any sort of document, including text files you just read in.

daveaglick commented 5 years ago

Shortcode support was just merged into develop from my working branch. The architecture is essentially done, now I just need to implement a few more actual shortcodes.

Here's where we landed:

By default, shortcodes are delimited with a syntax that also qualifies as XML processing instructions: <?# name args /?> or <?# name args ?>content<?#/ name ?> if the shortcode has content.
A new module, Shortcodes is responsible for parsing and evaluating shortcodes within documents.
The Shortcodes module has been added to the blog and docs recipes and is executed at the end of the rendering pipelines after Markdown, Razor, etc. has been processed.
Shortcode arguments are automatically split into key-value pairs to make writing shortcodes and interpreting their arguments easier and standardized.
I've implemented shortcodes for rendering metadata values (<?# meta key /?> or <?#= key /?> special syntax) and including files (<?# include file.html /?>). Many more to come prior to release.

daveaglick commented 5 years ago

I've got a decent set of initial shortcodes for the next version and will continue to add more. Just need to write some documentation and get the release out.

gep13 commented 5 years ago

@daveaglick sweet!! Looking forward to taking this for a spin!

daveaglick commented 5 years ago

@gep13 This issue kind of got hijacked by the general-purpose shortcode functionality. Please do open a new issue to specifically track a gallery shortcode if that’s something you’re still interested in.

statiqdev / Statiq.Web

Shortcodes #359

Content

Parameters

Evaluation Phases

Prerender Syntax

Postrender Syntax

Postrender Commented Syntax

Escaping Shortcodes

Nesting Shortcodes