search

hexojs / hexo

A fast, simple & powerful blog framework, powered by Node.js.
https://hexo.io
MIT License
39.2k stars 4.82k forks source link

{{ should not be rendered in a post #3259

Closed noraj closed 4 years ago

noraj commented 6 years ago

Environment Info

Node version(node -v): v10.9.0

Hexo and Plugin version(npm ls --depth 0): 

├── hexo@3.7.1
├── hexo-deployer-git@0.3.1
├── hexo-generator-archive@0.1.5
├── hexo-generator-category@0.1.3
├── hexo-generator-feed@1.2.2
├── hexo-generator-json-content@3.0.1
├── hexo-generator-tag@0.2.0
├── hexo-i18n@0.2.1
├── hexo-multiauthor@0.0.1
├── hexo-pagination@0.1.0
├── hexo-renderer-ejs@0.3.1
├── hexo-renderer-markdown-it@3.4.1
├── hexo-renderer-stylus@0.3.3
├── hexo-server@0.3.3
├── hexo-sliding-spoiler@1.1.6
├── hexo-tag-asciinema@0.0.1
└── hexo-tag-bootstrap@0.1.2

For BUG

I think I get this error message an hundred times:

Template render error: (unknown path) [Line 69, Column 34]
  unexpected token: }}
    at Object._prettifyError (/home/shark/Dev/hexo_rawsec/node_modules/nunjucks/src/lib.js:36:11)
    at Template.render (/home/shark/Dev/hexo_rawsec/node_modules/nunjucks/src/environment.js:524:21)
    at Environment.renderString (/home/shark/Dev/hexo_rawsec/node_modules/nunjucks/src/environment.js:362:17)
    at Promise (/home/shark/Dev/hexo_rawsec/node_modules/hexo/lib/extend/tag.js:66:9)
    at Promise._execute (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/debuggability.js:303:9)
    at Promise._resolveFromExecutor (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/promise.js:79:10)
    at Tag.render (/home/shark/Dev/hexo_rawsec/node_modules/hexo/lib/extend/tag.js:64:10)
    at Object.tagFilter [as onRenderEnd] (/home/shark/Dev/hexo_rawsec/node_modules/hexo/lib/hexo/post.js:230:16)
    at Promise.then.then.result (/home/shark/Dev/hexo_rawsec/node_modules/hexo/lib/hexo/render.js:65:19)
    at tryCatcher (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/home/shark/Dev/hexo_rawsec/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:693:18)
    at tryOnImmediate (timers.js:664:5)
    at processImmediate (timers.js:646:5)

Because I used {{ or }} in my markdown files. Because I'm writing technical stuff about templating languages for example. This is making the EJS template engine crash.

But instead of trying to compute all files blindly, Hexo should make the EJS engine aware of the markdown context.

I mean if there is }} or %} ok try to do something with it but if the same pattern is inside markdown inline or fenced block just ignore it, it's not for you silly EJS engine.

Same for other languages (other than markdown) and stuff inside a comment.

The EJS engine is blind, make him some eyes please.

Sometimes even using the {% raw %} tag I can't write the code I want to.

An additional mechanism can be to make a switch to turn off EJS parsing 1) globally in _config.yml and 2) locally inside the file frontammeter.

Not a marked problem. Marked render that as expected.

Related issue: #2030

curbengh commented 5 years ago

Try {% include_code %} with skip_render config as a workaround.

noraj commented 5 years ago

The only workaround that works is to include gist or external file.

9999years commented 5 years ago

I'm having the same issue on 3.8.0:

I have a file source/_posts/templates.md which contains text like so:

In Go's templates, blocks look like this: `{{block "template name" .}} (content) {{end}}`.

To which Hexo complains:

FATAL Something's wrong. Maybe you can find the solution here: 
http://hexo.io/docs/troubleshooting.html
Nunjucks Error:  [Line 1, Column 60] expected variable end
=====               Context Dump               =====
=== (line number probably different from source) ===
1 | <p>In Go's templates, blocks look like this: <code>{{block "template name" .}} (content)
2 | {{end}}</code>.</p>
3 |
=====             Context Dump Ends            =====

In Markdown sources (at least) we shouldn't render Nunjucks within literal blocks. Compare to John Gruber's introduction to the original Markdown, which states that "In a regular paragraph, you can create code span by wrapping text in backtick quotes. Any ampersands (&) and angle brackets (< or >) will automatically be translated into HTML entities. This makes it easy to use Markdown to write about HTML example code" -- even though Markdown supports HTML in the main body copy, the message is clear: text inside back-ticks (either inline as in `foo` or delimiting a multi-line code block) should not be messed with.

Now, how should we implement this? Perhaps in hexo-renderer-marked we could examine the output and manually escape {{ and {% within <code> tags, so that they're preserved once Nunjucks processes them; this might be as simple as replacing {{ with {% raw %}{{{% endraw %} and so on.

Any thoughts?

Edit: I tried patching hexo-renderer-marked, and by the time it gets to the file, it's already been run through Nunjucks, which is unfortunate. Maybe we can reorder that in lib/hexo/post.js?

noraj commented 4 years ago

This should work with any markdown renderer and not only marked. There should be a way to disable helper (and so nunjucks) via the frontmatter at least (low quality solution) or dynamically recognize that when {{ are in a markdown code block not to interprete them as nunjucks. Each time I want to write an article about Flask and Jinja2 my article is full of {{ and {% and that make it such a pain because of nunjucks and I don't even use any tag helper in the article. So yeah 1 year later it's still a big problem.

This is a complex thing to do because it has to be done after markdown to HTML rendering. Also so cases are trivial like <code>{{</code> but some looks more complex for example with <pre> in a <table> but in fact are not any harder. Let's considerate all <pre> like <code>.

Example of real-life nunjunks crash:

44 $ hexo generate
45 INFO  Start processing
46 FATAL Something's wrong. Maybe you can find the solution here: https://hexo.io/docs/troubleshooting.html
47 Nunjucks Error:  [Line 14, Column 114] unexpected token: }}
48     =====               Context Dump               =====
49     === (line number probably different from source) ===
50   9 | <h2 id="200-BoneChewerCon-Web"><a class="header-anchor" href="#200-BoneChewerCon-Web">🔗</a>200 - BoneChewerCon - Web</h2>
51   10 | <blockquote>
52   11 | <p>The devil is enticing us to commit some SSTI feng shui, would you be interested in doing so?</p>
53   12 | </blockquote>
54   13 | <p>By entering <code>{{ 7*7 }}</code> as a name, eg.<br>
55   14 | <a href="http://docker.hackthebox.eu:30402/?name=%7B%7B+7*7+%7D%7D">http://docker.hackthebox.eu:30402/?name={{+7*7+}}</a></p>
56   15 | <p><img src="https://i.imgur.com/KICwsir.png" alt=""></p>
57   16 | <p>we can see <code>49</code> displayed so the SSTI is effective.</p>
58   17 | <p>Now let's get real: <code>{{ config }}</code>:</p>
59   18 | <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">&lt;Config &#123;&#39;JSON_AS_ASCII&#39;: True, &#39;USE_X_SENDFILE&#39;: False, &#39;SESSION_COOKIE_SECURE&#39;: False, &#39;SESSION_COOKIE_PATH&#39;: None, &#39;SESSION_COOKIE_DOMAIN&#39;: None, &#39;SESSION_COOKIE_NAME&#39;: &#39;session&#39;, &#39;MAX_COOKIE_SIZE&#39;: 4093, &#39;SESSION_COOKIE_SAMESITE&#39;: None, &#39;PROPAGATE_EXCEPTIONS&#39;: None, &#39;ENV&#39;: &#39;production&#39;, &#39;DEBUG&#39;: False, &#39;SECRET_KEY&#39;: None, &#39;EXPLAIN_TEMPLATE_LOADING&#39;: False, &#39;MAX_CONTENT_LENGTH&#39;: None, &#39;APPLICATION_ROOT&#39;: &#39;&#x2F;&#39;, &#39;SERVER_NAME&#39;: None, &#39;PREFERRED_URL_SCHEME&#39;: &#39;http&#39;, &#39;JSONIFY_PRETTYPRINT_REGULAR&#39;: False, &#39;TESTING&#39;: False, &#39;PERMANENT_SESSION_LIFETIME&#39;: datetime.timedelta(31), &#39;TEMPLATES_AUTO_RELOAD&#39;: None, &#39;TRAP_BAD_REQUEST_ERRORS&#39;: None, &#39;JSON_SORT_KEYS&#39;: True, &#39;JSONIFY_MIMETYPE&#39;: &#39;application&#x2F;json&#39;, &#39;SESSION_COOKIE_HTTPONLY&#39;: True, &#39;SEND_FILE_MAX_AGE_DEFAULT&#39;: datetime.timedelta(0, 43200), &#39;PRESERVE_CONTEXT_ON_EXCEPTION&#39;: None, &#39;SESSION_REFRESH_EACH_REQUEST&#39;: True, &#39;TRAP_HTTP_EXCEPTIONS&#39;: False&#125;&gt;</span><br></pre></td></tr></table></figure>
60   19 | <p>Also by looking at the source we can request<br>
61     =====             Context Dump Ends            =====
62     at formatNunjucksError (/builds/rawsec/Rawsec-website/node_modules/hexo/lib/extend/tag.js:99:13)
63     at Promise.fromCallback.catch.err (/builds/rawsec/Rawsec-website/node_modules/hexo/lib/extend/tag.js:121:34)
64     at tryCatcher (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/util.js:16:23)
65     at Promise._settlePromiseFromHandler (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:547:31)
66     at Promise._settlePromise (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:604:18)
67     at Promise._settlePromise0 (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:649:10)
68     at Promise._settlePromises (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:725:18)
69     at _drainQueueStep (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:93:12)
70     at _drainQueue (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:86:9)
71     at Async._drainQueues (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:102:5)
72     at Immediate.Async.drainQueues [as _onImmediate] (/builds/rawsec/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:15:14)
73     at runCallback (timers.js:705:18)
74     at tryOnImmediate (timers.js:676:5)
75     at processImmediate (timers.js:658:5)
79 ERROR: Job failed: exit code 1

Here it was easy to spot because it was only a linked http://docker.hackthebox.eu:30402/?name=%7B%7B+7*7+%7D%7D (makrdown) auto-linked to <a href="http://docker.hackthebox.eu:30402/?name=%7B%7B+7*7+%7D%7D">http://docker.hackthebox.eu:30402/?name={{+7*7+}}</a></p> (html) where the text was automatically URL decoded, so I just had to put [http://docker.hackthebox.eu:30402/?name=%7B%7B+7*7+%7D%7D](http://docker.hackthebox.eu:30402/?name=%7B%7B+7*7+%7D%7D) (markdown) instead.

But re-tring now I have a fatal error which don't tell me where does it comes from:

FATAL Something's wrong. Maybe you can find the solution here: https://hexo.io/docs/troubleshooting.html
Template render error: (unknown path)
  Error: filter not found: attr
    at Object._prettifyError (/home/noraj/Dev/Rawsec-website/node_modules/nunjucks/src/lib.js:36:11)
    at /home/noraj/Dev/Rawsec-website/node_modules/nunjucks/src/environment.js:551:19
    at Template.root [as rootRenderFunc] (eval at _compile (/home/noraj/Dev/Rawsec-website/node_modules/nunjucks/src/environment.js:621:18), <anonymous>:67:3)
    at Template.render (/home/noraj/Dev/Rawsec-website/node_modules/nunjucks/src/environment.js:540:10)
    at Environment.renderString (/home/noraj/Dev/Rawsec-website/node_modules/nunjucks/src/environment.js:364:17)
    at Promise.fromCallback.cb (/home/noraj/Dev/Rawsec-website/node_modules/hexo/lib/extend/tag.js:120:48)
    at tryCatcher (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.fromNode.Promise.fromCallback (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/home/noraj/Dev/Rawsec-website/node_modules/hexo/lib/extend/tag.js:120:18)
    at Object.onRenderEnd (/home/noraj/Dev/Rawsec-website/node_modules/hexo/lib/hexo/post.js:291:22)
    at Promise.then.then.result (/home/noraj/Dev/Rawsec-website/node_modules/hexo/lib/hexo/render.js:79:21)
    at tryCatcher (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues (/home/noraj/Dev/Rawsec-website/node_modules/hexo/node_modules/bluebird/js/release/async.js:15:14)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)

which is far harder to spot.

curbengh commented 4 years ago

https://github.com/hexojs/hexo/issues/2622 https://github.com/hexojs/hexo/pull/2593#issuecomment-310879897

I'm thinking of parsing only {% in a post and skip {{, don't know if it's viable. a simple approach is to escape {{ to %7B%7B by default, but it's more of a workaround.

noraj commented 4 years ago

I'm thinking of parsing only {% in a post and skip {{

Why not if nunjucks is used only for tag helpers and tag helpers are using only {% but that won't resolve the problem when {% appears in code.

a simple approach is to escape {{ to %7B%7B by default, but it's more of a workaround.

This is not even a workaround, this will work only for the link case.

As I said earlier there are two solutions:

There should be a way to disable helper (and so nunjucks) via the frontmatter at least

like https://github.com/hexojs/hexo/pull/2593#issuecomment-310879897 suggested.

It is easier to implement but that require users to manually change their frontmatter when they encounter a conflict.

or dynamically recognize that when {{ are in a markdown code block not to interpret them as nunjucks.

Actually not detect in markdown but in rendered HTML so it will works with any renderer. This can be done with a regex or a HTML DOM parser to check if {{ or {% is present inside a <pre> or a <code>. So the workflow would be: markdown --(render)--> HTML --(nunjuncks patched to skip {{ or {%)--> HTML (with herlpers applied). For users, this would be automatic, will not require any change and will be retro-compatible but a little harder to implement.

noraj commented 4 years ago

PS : see here all the changes I needed to do to my post to make is pass the nunjucks logic. It cost me one hour. Only to properly escape everything and test and trying to get the output the less ugly possible. https://gitlab.com/rawsec/Rawsec-website/merge_requests/103/diffs?commit_id=8df837dfe6f7e7a2b19308b812979b9ded47cfe6

noraj commented 4 years ago

@curbengh is it planned to fix this bug that requires breaking changes for 5.0.0 milestone ?