0.60.0 goldmark: <html> indentation in partial/shortcode wrapped in `<pre><code>`

divinerites commented 4 years ago

What version of Hugo are you using (`hugo version`)?

$ hugo version
Hugo Static Site Generator v0.60.0/extended darwin/amd64 BuildDate: unknown

Does this issue reproduce with the latest release?

Yes. In lates 0.60.0 using goldmark

Following this discourse discussion, confirmed by @onedrawingperday , I fill a bug here
Using indentation in partial/shortcode (inline or classic) causes the HTML to be wrapped in code blocks
Option [markup.goldmark.renderer] unsafe = true has no effect.

bep commented 4 years ago

https://commonmark.org/help/tutorial/09-code.html

divinerites commented 4 years ago

Yes, I can understand that indenting with 4 spaces means a code block. No problem.

But I was not expecting that pure HTML/Go-template would be treated like this (it was not with blackFriday).

So, If i read between lines :-) this means that goldmark will treat every 4 spaces as a code block, even for HTML.

So a good prevention measure would be to change "tab->number of space" default in my editor (VSCode) to 3 (at least not 4).

Will look if it changes things. Thanks.

divinerites commented 4 years ago

Done some testing with 3 spaces.

If there is strictly 3 spaces as indentation, the result is OK : HTML is written as is.
At the moment where i do 2 indentations (2 x 3) => same problem than before

So I guess that at the moment any HTML code with several indentations (not equal to 3 exactly) will have a problem. I'll keep testing.

divinerites commented 4 years ago

So, after testing a bit I can confirm that :

HTML in a shortcode (inline or called via a partial) is processed by goldmark as if it was markdown.
So if you have ONE indentation of 3 (less than 4) space, it works as HTML. As before.
If you have 4 or more spaces (for ex 3+3 or 3+3+3) it is processed by goldmark and identify and rendered as code block.

I understand that Bep was not surprised by this behaviour, which is fine by me if it is expected behaviour. But I found this quite surprising because i would never expect HTML being impacted by indentation.

Workaround is to flat your HTML code if it could be called by shortcode. Not read friendly but efficient.

cc @onedrawingperday who helped me a lot on this diagnostic.

bep commented 4 years ago

OK, that seems to be a bug in Goldmark. I will create a bug report over there.

I understand that Bep was not surprised by this behaviour,

I didn't say I wasnt surprised, I just pointed to a possible explanation.

bep commented 4 years ago

OK; I had a look at this and I don't understand it. The example shows an inline shortcode using {{< which Goldmark should never see. I could understand it if the {{< was indented 4 chars, but that isn't the case. I tested some variants of this myself, and cannot reproduce.

jensamunch commented 4 years ago

Hi @bep - I'm not sure this should be closed. https://github.com/gohugoio/hugo/issues/6552

I made a screencast I think you might enjoy. Headless only works when the folder name does NOT match the name I'm calling it with 🤯

https://youtu.be/pRznYM-3Itk

divinerites commented 4 years ago

Later today, I'll provide access to the repo and prepare 3 pages with 3 different shortcodes/partials with this problem. Stay tuned.

divinerites commented 4 years ago

Bep & MooreReason, I invited you to the repo.

Use "Migration-en-goldmark" branch.
post vintage.fr.md is working fine (partial is flat HTML)
post test1.fr.md has problem (partial is fully indented HTML)
post test2.fr.md has same problem but different layout because indented a bit differently

First : thanks for your help, and let me know if I can do/test other things.

You can see the 3 post with exactly same code except HTML indentation on the 3 last post here : https://migration-en-goldmark--lagouille.netlify.com/dernieres-nouveautes/

divinerites commented 4 years ago

And test1/test2.html have 3 space indentation, and still get the problem. Mmmmmmmm

moorereason commented 4 years ago

@divinerites, Your issue appears to be related to the use of {{ .Content | markdownify }} in layouts/_default/single.html.

divinerites commented 4 years ago

Ahhhh. Yes makes sense. Thanks a lot. I Will take this in account.

So as the behaviour was different with use of blackFriday in 0.59.x, will look forward to see if this is expected behaviour or not and then change my code accordingly to the right practice. But anyhow i DO thanks you a lot for founding a "reasonable" cause & explanation :-). Thanks.

XhmikosR commented 4 years ago

I think this is the issue I'm hitting in Bootstrap too after updating Hugo.

This worked fine before but now renders invalid HTML:

<div class="bd-example">
  <table class="table">
    <thead>
      <tr>
        <th scope="col">Class</th>
        <th scope="col">Heading</th>
        <th scope="col">Heading</th>
      </tr>
    </thead>
    <tbody>
      <tr class="table-active">
        <th scope="row">Active</th>
        <td>Cell</td>
        <td>Cell</td>
      </tr>
      <tr>
        <th scope="row">Default</th>
        <td>Cell</td>
        <td>Cell</td>
      </tr>

      {{< table.inline >}}
      {{- range (index $.Site.Data "theme-colors") }}
      <tr class="table-{{ .name }}">
        <th scope="row">{{ .name | title }}</th>
        <td>Cell</td>
        <td>Cell</td>
      </tr>
      {{- end -}}
      {{< /table.inline >}}
    </tbody>
  </table>
</div>

ahmetb commented 4 years ago

Also hitting this after updating to v0.60.0. If you use {% %} syntax while invoking shortcodes, HTML tags inside the shortcodes are stripped off.

For example, shortcodes/details.html:

<details>
<summary>{{with .Get "title" }}{{.}}{{else}}Click to expand{{end}}</summary>
{{.Inner}}
</details>

Usage:

{{% details title="foo" %}}
bar
{{% /details %}}

Generated HTML Output:

<p>foo</p>
<!-- raw HTML omitted -->
<p>hello world</p>
<!-- raw HTML omitted -->

moorereason commented 4 years ago

@XhmikosR, how does that render?

@ahmetb, are you enabling unsafe in Goldmark?

XhmikosR commented 4 years ago

@moorereason <pre><code> inside the HTML with unclosed div tags. I'll be able to provide the exact snippet tomorrow since for now I just reduced indentation to work around this issue https://github.com/twbs/bootstrap/pull/29750/files#diff-dcbb1aeb2504a4b02de5c00ba0296ceb

ahmetb commented 4 years ago

@ahmetb, are you enabling unsafe in Goldmark?

First time I'm hearing about Goldmark and the goldmark: configuration.

I don't have markdown: key in my config.yaml and actually things work fine when I downgrade to v0.59.1. Was I relying on undocumenting behavior previously?

ahmetb commented 4 years ago

...to add to my previous comment:

I did not realize Blackfriday was replaced with Goldmark.

My Blackfriday config also did not have any relevant keys with respect to unsafe html.

XhmikosR commented 4 years ago

I tried this without the workaround and it renders to:

HTML

```html


</tbody>

Class	Heading	Heading
Active	Cell	Cell
Default	Cell	Cell
Primary	Cell	Cell
Secondary	Cell	Cell
Success	Cell	Cell
Danger	Cell	Cell
Warning	Cell	Cell
Info	Cell	Cell
Light	Cell	Cell
Dark	Cell	Cell

```

divinerites commented 4 years ago

This is why we use a workaround :-)

Without it is render as pre/code block.

hucste commented 4 years ago

I have quasy-same problem with v0.6x.

See my markup config:

[markup]
    defaultMarkdownHandler = "goldmark"
    [markup.blackFriday]
    angledQuotes = false
    footnoteAnchorPrefix = ""
    footnoteReturnLinkContents = ""
    fractions = true
    hrefTargetBlank = false
    latexDashes = true
    nofollowLinks = false
    noreferrerLinks = false
    plainIDAnchors = true
    skipHTML = false
    smartDashes = true
    smartypants = true
    smartypantsQuotesNBSP = false
    taskLists = true
    [markup.goldmark]
    [markup.goldmark.extensions]
      definitionList = true
      footnote = true
      linkify = true
      strikethrough = true
      table = true
      taskList = true
      typographer = true
    [markup.goldmark.parser]
      attribute = true
      autoHeadingID = true
    [markup.goldmark.renderer]
      hardWraps = false
      unsafe = true
      xHTML = false
    [markup.highlight]
    codeFences = true
    hl_Lines = ""
    lineNoStart = 1
    lineNos = false
    lineNumbersInTable = true
    noClasses = true
    style = "monokai"
    tabWidth = 4
    [markup.tableOfContents]
    endLevel = 3
    startLevel = 2

shortcode code: http://doc.huc.fr.eu.org/en/web/hugo/hugo-shortcodes/#code
shortcode file: http://doc.huc.fr.eu.org/en/web/hugo/hugo-shortcodes/#file (for this site's version, I use actually v0.59.1)
RAW src shortcode code: https://framagit.org/sh-web/hugo/doc.huc.fr.eu.org/raw/master/layouts/shortcodes/code.html
RAW src shortcode file: https://framagit.org/sh-web/hugo/doc.huc.fr.eu.org/raw/master/layouts/shortcodes/file.html

With v0.6(0-4).*, for thoses shortcodes, HTML code disappear.

I push thoses images: v0 6x_error-shortcode-code_2019-12-12_23-51-36 v0 6x_error-shortcode-file_2019-12-12_23-52-54

hucste commented 4 years ago

I confirm this bug always with 0.63.x, and 0.64.x too! :(

satotake commented 4 years ago

@hucste v0.65 improves your results.

スクリーンショット 2020-02-24 13 36 09

and tweaking a little, hugo works perfectly.

スクリーンショット 2020-02-24 13 40 07

I just set a valid language as second arg in this line

satotake commented 4 years ago

After struggling, I reached quite a simple solution for this issue in terms of commonmark's spec. Remove blank lines if you want to use html tags inside commonmark. As long as blank lines are not inserted into html part, you do not need to flat your partial html.

For example, in @XhmikosR 's case, this PR will fix it.

This behavior seems to be commonmark's spec rather than Hugo's issue. I created a minimal example with commonmark.js

hucste commented 4 years ago

OK, I understand.

@satotake: With your explains, now it runs correctly.

And I test with oldiers 0.6x, it's OK, too.

Thanks for all! :D

PS: I think, you can close this ticket! ;)

jmooring commented 3 years ago

Closing. The only way to reproduce the original problem is by erroneously piping .Content through markdownify as noted by @moorereason in https://github.com/gohugoio/hugo/issues/6553#issuecomment-559863928.

Simplified test site:

git clone --single-branch -b hugo-github-issue-6553 https://github.com/jmooring/hugo-testing hugo-github-issue-6553
cd hugo-github-issue-6553
hugo server

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

gohugoio / hugo