Custom highlight function doesn't respect whitespace in code blocks

michaeloliverx commented 3 years ago

HTML Example

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Example HTML5 Document</title>
  </head>
  <body>
    <p>Test</p>
  </body>
</html>

<img width="523" alt="image" src="https://user-images.githubusercontent.com/55017335/111953573-b93baa00-8ade-11eb-8580-3e98e8ba183d.png">

## Python Example

from typing import Optional

from fastapi import FastAPI
from pydantic import BaseModel

class Item(BaseModel):
    name: str
    description: Optional[str] = None
    price: float
    tax: Optional[float] = None

app = FastAPI()

@app.put("/items/{item_id}")

async def create_item(item_id: int, item: Item, q: Optional[str] = None):

    result = {"item_id": item_id, **item.dict()}
    if q:
        result.update({"q": q})
    return result


<img width="709" alt="image" src="https://user-images.githubusercontent.com/55017335/111953896-2d764d80-8adf-11eb-8d2a-3be746051a6a.png">
---

## UPDATE
As a workaround I got `shiki` working with the following snippet:
````js
const shiki = require("shiki");

const escape_svelty = (str) =>
  str
    .replace(
      /[{}`]/g,
      (c) => ({ "{": "&#123;", "}": "&#125;", "`": "&#96;" }[c])
    )
    .replace(/\\([trn])/g, "&#92;$1");

async function highlighter(code, lang) {
  const highlighter = await shiki.getHighlighter({ theme: "github-dark" });
  const highlightedCode = escape_svelty(
    highlighter.codeToHtml(code, lang || "text")
  );
  console.log(highlightedCode);
  return `{@html \`${highlightedCode}\` }`;
}

module.exports = {
  extensions: [".svx", ".md"],
  highlight: {
    highlighter: highlighter,
  },
};

It gives me what I want:

I don't know the stance MDsveX should take on code blocks.. having flexibility is king and I love being able to customise code output but I think there needs to be more information in the docs about this.

pngwn commented 3 years ago

I wonder why this is happening. Will look into it.

jthegedus commented 2 years ago

@michaeloliverx thanks for sharing your code snippet, it gave me a nice starting point. Sharing my config here as I am consuming the escapeSvelte function from the mdsvex library itself simplifying things a bit:

// mdsvex.config.js
import shiki from "shiki";
import { escapeSvelte } from "mdsvex";

const config = {
  "extensions": [".svelte.md", ".md", ".svx"],

  highlight: {
    highlighter: async (code, lang = "text") => {
      const highlighter = await shiki.getHighlighter({ theme: "github-dark" });
      const highlightedCode = escapeSvelte(highlighter.codeToHtml(code, lang));
      return `{@html \`${highlightedCode}\` }`;
    },
  },

  "smartypants": {
    "dashes": "oldschool",
  },

  "remarkPlugins": [],
  "rehypePlugins": [],
};

export default config;

michaeloliverx commented 2 years ago

@jthegedus I am glad it helped! Since then I have been using shiki-twoslash its really nice!

Take the following markdown:

```ts twoslash
type Post = {
  title: string;
  description: string;
};

function getPosts(): Array<Post>{
  return []
}

const posts = getPosts();


Generates code samples with typescript hints on mouse hover:

<img width="655" alt="image" src="https://user-images.githubusercontent.com/55017335/136346839-ef756509-d178-43f6-8925-f10a8d09d71a.png">

You can also highlight lines with the following syntax:

````markdown
```ts twoslash
```ts twoslash {1-4}
type Post = {
  title: string;
  description: string;
};

function getPosts(): Array<Post>{
  return []
}

const posts = getPosts();

It works on non JS/TS code samples too just omit the twoslash directive. You can even add custom properties for example a title or filename:

```ts {1-4} filename="some-file.ts"
type Post = {
  title: string;
  description: string;
};
...


They will be added to the `pre` HTML element as attributes where you could choose to display them using CSS:

<img width="454" alt="image" src="https://user-images.githubusercontent.com/55017335/136347971-5f64ccef-2e56-444e-9db2-f2c62b6f30e3.png">

Anyway enough shilling here is my config file:

```js
// mdsvex.config.js

/**
 * Full MDsveX Options Documentation:
 * https://mdsvex.com/docs#options
 */

import { lex, parse } from "fenceparser";
import { renderCodeToHTML, runTwoSlash, createShikiHighlighter } from "shiki-twoslash";

/** @type {Parameters<typeof import("mdsvex").mdsvex>[0]} */
export const config = {
  extensions: [".svx"],
  highlight: {
    async highlighter(code, lang, meta) {
      // Adapted from the `remark-shiki-twoslash` repo
      // See: https://github.com/shikijs/twoslash/blob/fbf061261fcda90c46e946ce1e2e9357d465c145/packages/remark-shiki-twoslash/src/index.ts#L172-L215
      let fence;

      try {
        fence = parse(lex([lang, meta].filter(Boolean).join(" ")));
      } catch (error) {
        throw new Error(`Could not parse the codefence for this code sample \n${code}`);
      }

      let twoslash;
      if (fence.twoslash === true) {
        twoslash = runTwoSlash(code, lang);
      }

      const highlighter = await createShikiHighlighter({ theme: "github-dark" });
      const html = renderCodeToHTML(code, lang, fence, {}, highlighter, twoslash);
      return `{@html \`${html}\` }`;
    },
  },
};

You will also need some additional CSS, you can find a base under point 3 here: https://github.com/shikijs/twoslash/tree/main/packages/remark-shiki-twoslash#plugin-setup

jthegedus commented 2 years ago

@michaeloliverx Thanks for pointing me to twoslash, very cool project :pray:

benmccann commented 3 weeks ago

I can't reproduce this. Is it still an issue?

pngwn / MDsveX

Custom highlight function doesn't respect whitespace in code blocks #212

HTML Example