feat: migrate to shikiji

[antfu]

Benefits:

• Smaller client payload size, as the code is no longer duplicated in HTML (https://github.com/antfu/shikiji#lightdark-dual-themes)
• Shikiji is hast based, which makes it much easier to transform and customize. A lot of things can be done more robustly without RegExp. shikiji-transformers is created to mimic the behavior of shiki-processor. It also allows users to provide their own transformers.
• More efficient langs/themes loading based on ESM rather than fs
• Install size: 17MB -> 9MB

This should be quite compatible with the previous behavior. The only change users might notice is that .vp-code-dark and .vp-code-light are no longer used, instead, a single .vp-code takes over.

In general, I think this PR is good to do, marking it as draft as I need to handle some details on shikiji side and release a stable version before this gets merged. Should be ready :)

[brc-dd]

Still need a way to specify aliases. Earlier this hack used to work:

// .vitepress/config.ts
import { BUNDLED_LANGUAGES } from 'shiki'

(BUNDLED_LANGUAGES.find(lang => lang.id === 'foo').aliases ??= []).push('bar')

export default defineConfig({
  // ...
})

Can this be supported in shikiji via some config?

[brc-dd]

Another thing, is the syntax for adding custom languages still something like?

    languages: [
      {
        id: 'vue-vine',
        scopeName: 'source.vue-vine',
        path: join(
          __dirname,
          './vine-ts.tmLanguage.json',
        ),
        embeddedLangs: [
          'vue-html',
          'css',
          'scss',
          'sass',
          'less',
          'stylus',
        ],
      },
    ],

Also, it seems if someone has specified languages, it won't be merged with bundledLanguages 👀

[brc-dd]

Rest looks good to me. I'm fine with introducing some breaking changes this once, as long as we can mention the migration path here, so that we can just redirect users to visit this PR for solutions.

[antfu]

Another thing, is the syntax for adding custom languages still something like?

Yes, it support custom langs. But path option is not, as shikiji does not have fs access. Users need to load by themselves and pass the json. I will update the docs in shikiji and mention about that.

Edit: Updated https://github.com/antfu/shikiji/blob/main/docs/languages.md#migrate-from-shiki

[antfu]

Should be good to go now

[brc-dd]

Just curious,

const vineGrammar = JSON.parse(fs.readFileSync(join(__dirname, './vine-ts.tmLanguage.json'), 'utf8'))

how is this different from import vineGrammar from './vine-ts.tmLanguage.json' 👀

[antfu]

No, both would work. Tho in real ESM you can't directly import .json

[brc-dd]

Yeah I meant with assert { type: 'json' } (or the with syntax once node supports that 😅)

[brc-dd]

Also, in https://github.com/antfu/shikiji/blob/main/docs/languages.md#migrate-from-shiki

I don't think id exists on that type. Needs to be removed from the second block?

For vue-vine guys, they need to change name to vue-vine in their language file. Looks like shikiji uses name as the id too? Other stuff seems to be working fine.

[brc-dd]

released in v1.0.0-rc.30 🎉

[husayt]

The way we used to import lang definitions from shiki was through tmLanguage.json files in languages folder in shiki dist.

 import htmlLang from "shiki/languages/html.tmLanguage.json" assert { type: "json" };

I noticed there are no *.tmLanguage.json with shikiji.

What shall we use here?

[brc-dd]

@husayt Hi, what's your use case for that?

[husayt]

it is for me to add more aliases

export const markdown: MarkdownOptions = {
languages: [
    {
      grammar: htmlLang,
      id: "myHtml",
      scopeName: "text.html.basic",
      aliases: ["html:preview", "html:preview:expanded:no-codepen"],
    } as any,
  ],
};

[brc-dd]

Can you try if this works:

export const markdown: MarkdownOptions = {
  languageAlias: {
    'myHtml': 'html',
    'html:preview': 'html',
    'html:preview:expanded:no-codepen': 'html',
  }
} 

[elringus]

@brc-dd @antfu Can you please clarify how to migrate the following (proposed migration guides don't cover types and themes):

import { defineConfig, MarkdownOptions } from "vitepress";
import { ILanguageRegistration } from "shiki";

export default defineConfig({
    ...
    markdown: md
});

const md: MarkdownOptions = {
    languages: [mylang],
    theme: {
        light: "./mylang-theme-light.json",
        dark: "./mylang-theme-dark.json"
    }
};

const mylang: ILanguageRegistration = {
    id: "mylang",
    aliases: ["mylang-alias"],
    scopeName: "source.mylang",
    path: "./mylang-textmate.json"
};

[antfu]

import { defineConfig } from "vitepress";
import { MarkdownOptions } from "vitepress";
import { LanguageRegistration } from "shikiji";

import grammar from "./mylang-textmate.json"
import themeDark from "./mylang-theme-dark.json"
import themeLight from "./mylang-theme-light.json"

export default defineConfig({
    ...
    markdown: md
});

const md: MarkdownOptions = {
    languages: [mylang],
    theme: {
        light: themeDark,
        dark: themeLight,
    }
};

const mylang: LanguageRegistration = {
    id: "mylang",
    aliases: ["mylang-alias"],
    scopeName: "source.mylang",
    ...grammar,
};

[elringus]

Thank you, that seem to be working fine. Had type errors for imported grammar and theme jsons, but I didn't type-check them previously, so it's probably an issue on my side. At runtime everything seem to work same as before.