IAL removed from fenced code blocks instead of rendered verbatim

[RobertDober]

As discovered in https://github.com/elixir-lang/ex_doc/pull/1400 fenced codeblocks should render IAL annotation (and also annotations, to be checked) verbatim.

However the IAL annotation disappears.

[RobertDober]

@josevalim @milmazz I will push this to 1.5.0 as feared.

I have added this special case for the ex_doc PR and now I am afraid I need more context when rendering blockquotes and headlines inside code blocks.

If and when 1.5.0 is ready this should be quite simple to implement though as there will be no scanning inside code blocks anymore (José's lookahead removed parallel scanning and therefore I will always scan only as much as needed and have a nice context)

[josevalim]

@RobertDober there is absolutely no rush from our side. :) I will also send a PR hopefully soon to preserve spaces inside HTML tags, so it should provide another option for tackling this.

[milmazz]

I have added this special case for the ex_doc PR and now I am afraid I need more context when rendering blockquotes and headlines inside code blocks.

What kind of context do you need? AFAIK the only case needed by ex_doc is the following:

```
> ### my title {: .warning}
>
> my description
```

And we should translate that to something like:

<blockquote>
  <h3 class="warning">my title</h3>
  <p>my description</p>
</blockquote>

And you already covered this case :)

[josevalim]

I think he meant more context at the implementation level. The code needs to know when not to extract IAL. :)

[RobertDober]

meant more context at the implementation level. The code needs to know when not to extract IAL. :)

Absolutely and maybe the fix is not that hard but I have lost myself in how 1.4.x is written, 1.5 will be my first own implementation, this is still in essence Dave's code and that was written a long time ago, very fast, for something needed badly, by a genius, so I forgive myself for being kinda lost....

[RobertDober]

As in real life I am talking too much, the information I needed to convey goes here :blush: @josevalim please base your PR against branch rel-1.4.20, master is broken, however if you have not worked on the space preserving PR I have some time to do that, what space do you mean?

Here are cases that do that already

iex(17)> markdown="<pre><code>           hello</code></pre>"
"<pre><code>           hello</code></pre>"
iex(18)> EarmarkParser.as_ast(markdown)
{:ok, [{"pre", [], ["<code>           hello</code>"], %{verbatim: true}}], []}
iex(19)> markdown="""
...(19)> <pre>
...(19)> <code>
...(19)>    hello
...(19)> </code>
...(19)> </pre>
...(19)> """
"<pre>\n<code>\n   hello\n</code>\n</pre>\n"
iex(20)> EarmarkParser.as_ast(markdown)
{:ok, [{"pre", [], ["<code>", "   hello", "</code>"], %{verbatim: true}}], []}

My guess would be that this case is bugging you...

iex(21)> markdown="<code>   hello</code>"
"<code>   hello</code>"
iex(22)> EarmarkParser.as_ast(markdown)
{:ok, [{"code", [], ["hello"], %{verbatim: true}}], []}